CROSSREFERENCE TO RELATED APPLICATIONS
This application claims the benefit of U.S. provisional application No. 62/733,425 titled “Augmented Reality Mathematics in Learning Platforms” filed on Sep. 19, 2018, which is herein incorporated by reference in its entirety for all purposes.
TECHNICAL FIELD
The following relates to automatically providing mathematical information related to a point or object of interest overlaid onto a video feed of a location or area in which the point or object exists to users of an online learning platform.
BACKGROUND
Augmented reality systems supplement reality, in the form of a captured image or video stream, with additional information. In many cases, such systems take advantage of a portable electronic device's imaging and display capabilities and combine a video feed with data describing objects in the video. In some examples, the data describing the objects in the video can be the result of a search for nearby points of interest.
While many types of supplemental information for a point or object of interest are available, none provide mathematical information. There is a need for an online learning platform that provides mathematical information for points or objects of interest.
For a better understanding of the various described implementations, reference should be made to the Detailed Description below, in conjunction with the following drawings.
Like reference numerals refer to corresponding parts throughout the drawings and specification.
DETAILED DESCRIPTION
Reference will now be made in detail to various embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, wellknown methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.
The education platform 110 is communicatively coupled to client device 130 via a network 140. A user 101 may access education platform 110 using one or more client devices 130. A client 130 accesses digital content from education platform 110 through network 140 and presents digital content to user 101. In some embodiments, client device 130 is an augmented reality enabled device that enables a user 101 to have an immersive experience navigating the personalized mathematical content and activities. Client 130 may include software, such as an augmented reality applied mathematics application (not shown) for rendering digital mathematical content and related activities received from platform 110. The augmented reality applied mathematics application may execute on the client device 130 or in a browser application that executes on the client device 130.
Network 140 enables communications among the entities connected to them through one or more localarea networks and/or widearea networks. In one embodiment, network 140 is the Internet and uses standard wired and/or wireless communications technologies and/or protocols. Data exchanged over network 140 can be represented using technologies and/or formats including hypertext markup language (HTML), extensible markup language (XML), and/or JavaScript Object Notation (JSON). In addition, all or some of the transmitted data can be encrypted using conventional encryption technologies such as the secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), and/or Internet Protocol security (IPsec). In another embodiment, the entities use custom and/or dedicated data communications technologies instead of, or in addition to, the ones described above.
Education platform 110 stores mathematical educational content items and serves these items to users of client devices 130 in accordance with some implementations. In the illustrated embodiment, the education platform 110 includes a mathematical content repository 120, a user information repository 125, and a mathematical educational content and learning activities engine 115, referred to as the mathematical education engine 115 hereafter. In some implementations (not illustrated), content repository 120 or a portion thereof, is provided by a thirdparty (not shown) and may be communicatively networked with mathematical education engine 115, such as, via network 140.
Content in mathematical content repository 120 includes a plurality of mathematical content records, which are discussed further with reference to
Mathematical education engine 115 provides gamified personalized mathematical learning information and activities to users of education platform 110. Mathematical education engine 115 includes a content processing module 160, a user learning profile module 170, and a mathematical augmentation module 180.
Content processing module 160 processes content captured by user device 130 (e.g., captured video streams, images, etc., such as captured image 205) as well as user interactions with mathematical content and activities. User learning profile module 170 generates user records for each user of education platform 110, and stores these user records in the user information repository 125, which is discussed further with reference to
Mathematical augmentation module 180 generates gamified personalized mathematical learning information and activities for users of education platform 110. In some implementations, mathematical augmentation module 180 employs Artificial Intelligence techniques such as machine learning and iterative learning. Examples of such techniques include, but are not limited to, expert systems, casebased reasoning, Bayesian networks, behaviorbased AI, neural networks, fuzzy systems, evolutionary computation (e.g., genetic algorithms), swarm intelligence (e.g., ant algorithms), and hybrid intelligent systems (e.g., expert inference rules generated through a neural network or production rules from statistical learning).
Many conventional features, such as firewalls, load balancers, application servers, failover servers, network management tools and so forth are not shown so as not to obscure the features of the system. A suitable service for implementation of the education platform is the CHEGG® service, found at www.chegg.com; other education platform services are known as well, and can be adapted to operate according to the teaching disclosed here. The term “service” in the context of the education platform 110 represents any computer system adapted to serve content using any internetworking protocols and is not intended to be limited to content uploaded or downloaded via the Internet or the HTTP protocol. The term “module” refers to computer program logic for providing a specified functionality. A module can be implemented in hardware, firmware, and/or software. A module is typically stored on a computerreadable storage medium such as storage device, loaded into a memory, and executed by a processor. In general, functions described in one embodiment as being performed on the server side can also be performed on the client side in other embodiments if appropriate. In addition, the functionality attributed to a particular component can be performed by different or multiple components operating together.
When the user captures an image using a camera of the augmented reality device 130, one or both of the mathematical education engine 115 and augmented reality applied mathematics application displays mathematical information pertaining to one or more objects of interest in the captured image. The augmented mathematical information is superimposed over the one or more objects of interest in the captured image. Examples of mathematical information include geometrical shape(s) that constitute the object of interest, formula associated with the geometrical shape, other mathematical information such as mathematical concepts, and educational activities associated with the geometrical shape.
In this illustrated embodiment, one or more of the augmented reality applied mathematics application and mathematical education engine 115 has overlaid the following information onto the image 205: (i) a triangle 230 over the dome of the U.S. Capitol Building, where the triangle 230 has three vertices represented by markers M1, M2 and M3; (ii) supplemental mathematical information 235 corresponding to the triangle 230; (iii) supplemental mathematical information 240 corresponding to the U.S. Capitol Building; (iv) a rectangle 250 over a bus driving by the U.S. Capitol Building, where the rectangle 250 has four vertices represented by markers M4M8; and (v) supplemental mathematical information 255 corresponding to the bus.
As illustrated, supplemental mathematical information 235 corresponding to the triangle 230 includes the name of the shape (“Right Triangle”), either one or more formulas for right angle triangles, or links thereto, a “Learn More” feature that enables a user to access additional information about right angle triangles and possibly related shapes, and learning activities associated with the shape that are mathematical in their content. The additional information might also include size, lengths of various sides, of the right angle triangle 230 that has been superimposed on image 202. Examples of learning activities associated with the right triangle shape might be a game in which the user has to classify triangles, draw triangles, etc., or one in which the user watches a video about triangles, etc. Supplemental mathematical information 240 corresponding to the U.S. Capitol Building includes a “Learn More” feature that enables a user to access additional information about the U.S. Capitol building, and learning activities associated with the U.S. Capitol Building that are mathematical in their content. Examples of mathematical learning activities associated with the U.S. Capitol Building include a video about various mathematical proportions in its architecture. Supplemental mathematical information 255 corresponding to the bus includes a “Learn More” feature that enables a user to access additional information about laws of motion of physics, and learning activities associated with the laws of motion that are mathematical in their content.
As discussed later with reference to
At block 310, the method starts when the augmented reality applied mathematics application detects that a video stream has been captured by device 130. User 101 uses device 130 to capture a scene, such as a building, a tree, road, a painting, a sports scene, etc. Although described here in reference to a video stream, another embodiment of the disclosed technology includes capturing and displaying a single still image or a series of still images.
At block 320, the augmented reality applied mathematics application optionally detects or obtains one or more of geographic position (e.g., GPS data), camera direction, and/or tilt of device 130.
At block 330, the augmented reality applied mathematics application processes the detected information (if obtained, at block 320) and video stream or image (at block 310) to determine objects/points of interest that occur in the captured video stream or image. For example, GPS information for the device 130 may indicate that the device 130 is in the vicinity of the U.S. Capitol Building. This indication is used to then filter the possibilities of images in the captured video stream to the U.S. Capitol Building. In some embodiments, this processing is performed in whole or in part by the mathematical augmentation module 180. In other embodiments, the processing is based solely on the captured image itself and used to determine the objects/points of interest that occur in the captured video stream or image.
A point or object of interest (referred hereafter as “point of interest”) can be any object that can be viewed in a captured image and that can be decomposed into one or more geometrical shapes. A point of interest can be a place, building, structure, object, etc. and can be stationary or mobile. A point of interest may exist in various environments or contexts, such as in architecture, arts, sports, physics, nature, biology.
For example, in the architectural context, a point of interest captured by a client device 130 may include a building, which may be decomposed into geometric shapes and used to display augmented mathematical information. An example is provided in
In the arts context, a point of interest captured by a client device 130 may include a painting, which may be decomposed into geometric shapes, and used to display augmented mathematical information.
In the sports context, a point of interest captured by a client device 130 may include various actions in sporting events such as games, which may be decomposed into geometric shape(s) (such as, trajectories of motion) and used to illustrate Newton's laws, laws of movement, inertia, acceleration, etc.
At block 340, the augmented reality applied mathematics application and/or the education engine 115 determine mathematical information for the object of interest that have been deemed to occur in the captured video stream. In some embodiments, a lookup is performed of the mathematical content repository 120, which stores raw material for the mathematical information as content records 500 and is discussed further with reference to
In some embodiments, optionally, at block 345, the mathematical educational information is personalized based on a user profile stored in user record, which is discussed further with reference to
At block 350, the augmented reality applied mathematics application superimposes the mathematical information (from block 340) over the one or more objects of interest in the captured image. An example is provided in
In some embodiments, at block 350, augmented mathematical information is superimposed or overlaid on the captured image. In some embodiments, first mathematical information (e.g., geometric shapes) is superimposed on the point of interest. For example, referring to
Further, in some embodiments, different combinations and permutations of augmented mathematical information (e.g., geometric shapes) are superimposed or overlaid on the captured image. It can be understood that any object can be seen as being composed of many different combinations of geometric shapes. For example, a rectangle can be decomposed into constituent rectangles, or constituent triangles, or constituent triangles and rectangles. Accordingly, a user may be first presented with first augmented mathematical information, but presented with a different augmented mathematical information at a later time. The second presentation may be in response to receiving a user request, or may be made automatically, e.g., after a certain amount of time has passed.
At block 360, the augmented reality applied mathematics application may enable user 101 to select, view, and interact with the superimposed mathematical information to perform a variety of actions.
One user action is manipulation of geometrical shape(s). One example of user manipulation is rotation, expansion or contraction of the geometric shape. Referring to
One example of user manipulation is zooming in/out of the geometric shape. For example, a user 101 may be able to zoom into a complex geometrical shape such as a polygon to view more granular shapes (e.g., triangles) that constitute the polygon.
Another feature that may be available is a search feature that allows a user to enter a search term (e.g., “triangle”) in a search bar (not shown) to conduct a search for a specific geometric shape. In some embodiments, one or both of the mathematical education engine 115 and augmented reality applied mathematics application determines the existence of the search term (e.g., “triangle”) in the captured image 205 and displays the search results, if any.
Another user action is a doityourself feature that enables a user to discover geometric shapes themselves in a captured image. Accordingly, the user may provide input of geometric shapes, e.g., using their finger or stylus, or using hand gestures. In some embodiments, one or both of the mathematical education engine 115 and augmented reality applied mathematics application processes the user input to display the userinput shape and provides feedback and associated supplemental mathematical information for the geometrical shape(s). As an example, referring to
In some embodiments, the doityourself feature is spurred on by one or both of the mathematical education engine 115 and augmented reality applied mathematics application. Accordingly, the one or both of the mathematical education engine 115 and augmented reality applied mathematics application provide some information about a shape, such as, a formula (e.g., surface area for a triangle), or name of shape (e.g., right angle triangle), and ask the user to find the corresponding shape (e.g., right angle triangle) in the captured image (e.g., 202) and draw it on the captured image. An example of a user prompt might be: “Find and draw a right angle triangle in this scene.” Further, optionally, dimensions may be provided. An example of a user prompt might be: “Find and draw a circle with 2 cm radius in this scene.” In yet another implementation of the doityourself feature, the user is asked to find and draw a shape on the captured image in a timed manner. An example of such a command might be “Find and draw a circle with 2 cm radius in 20 seconds in this scene.” These are examples of gamifying mathematics for a user and may help the user learn to identify and draw geometric shapes and mathematical concepts.
Another user action is a multiuser mode, in which two or more users who are capturing the same scene (e.g., 202) are prompted to perform mathematical activities in a collaborative, exploratory, or competitive manner. Accordingly, upon determining that two or more users are in the same location and capturing the same scene, the users may be prompted to compete with each other in identifying geometric shapes. An example of a user prompt that promotes competition might be: “Can you find and draw a right angle triangle in this scene faster than User X?”
In some embodiments, a learning activity 570 is one of passive, active, and recall. Activities are defined as “passive” when a user interacts with already created content, such as when a user looks at a video for a Newton's laws. Activities are defined as “active” when a user creates new own user generated content, such as, personal notes, highlights, citations, and other comments, and connects and exchanges feedback with peers, among others. Recall activities test a user against knowledge acquired from passive and active activities. In some cases, recall activities are used for evaluating student performance in the context of an educational course, and may include homework assignments, tests, quizzes, and the like.
Referring again to
A user record 400 may include: a unique record identifier 410 that identifies the particular user record; identification information 415 for the user, such as, the user's name, email address, age, address, mobile device number, etc.; educational biographical information 520; and historical access information 430 including records of user's activities on the educational platform 110.
Educational biographical information 420 may include historical and current biographical information, such as universities attended by the user, courses taken, grades in those courses, courses currently registered for, major(s) declared, degree(s) obtained, degree(s) user wishes to obtain, and so on. Educational biographical information 420 may also include a calendar of user's personal, social, and educational commitments, such as upcoming assignment deadlines, upcoming exam deadlines, etc.
Historical access information 430 indicates which content in mathematical content repository 120 has been accessed by user 101. In some embodiments, historical access information 430 indicates content that has accessed by the user while viewing augmented content, such as illustrated in
A user record 400 may further include a learning profile 440 for user 101. Learning profile 440 may indicate one or more preferred modes of learning for user 101 and may indicate preferences for: type of activity preferred (e.g., active, passive, or recall), type of content (e.g., video, lecture, book, etc.), duration of activity (short vs. long), and so on. For example, one user may learn better by watching videos, while another may learn better by reading text. In another example, one user may learn better if learning sessions are frequently interspersed with nonlearning recreational sessions, while another may learn better with long undisturbed sessions. In another example, one user may learn better by repetition or refreshing of previously learned material, while another may prefer mostly or all new material. In yet another example, user 101 may have different preferred modes of learning for different subjects, courses, topics within a subject or course, or even concepts within a subject or course. In yet another example, user 101 may have different preferred modes of learning at different times. For example, at the beginning of an academic term, user 101 may prefer a first mode of learning (such as, a recall activity comprising refresh of material learned in a preceding class, use of a lot of exercises to learn new topics), and at the end of an academic term, user 101 may prefer a second mode of learning (such as, a recall activity comprising refresh of material learned in current class).
Memory 606 includes volatile and/or nonvolatile memory. Memory 606 (e.g., the nonvolatile memory within memory 606) includes a nontransitory computerreadable storage medium. Memory 606 optionally includes one or more storage devices remotely located from the processors 602 and/or a nontransitory computerreadable storage medium that is removably inserted into the server system 600. In some embodiments, memory 606 (e.g., the nontransitory computerreadable storage medium of memory 606) stores the following modules and data:

 an operating system 608 that includes procedures for handling various basic system services and for performing hardwaredependent tasks;
 a network communication module 610 that is used for connecting the education engine 115 to other computing devices via one or more network interfaces 604 connected to one or more networks 140 (
FIG. 1 );  content processing module 160 or a portion thereof;
 user learning profile module 170 or a portion thereof; and
 mathematical augmentation module 180 or a portion thereof.
Each of the modules stored in memory 606 corresponds to a set of instructions for performing one or more functions described herein. Separate modules need not be implemented as separate software programs. The modules and various subsets of the modules may be combined or otherwise rearranged. In some embodiments, memory 606 stores a subset or superset of the modules and/or data structures identified above.
The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the scope of the claims to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen in order to best explain the principles underlying the claims and their practical applications, to thereby enable others skilled in the art to best use the embodiments with various modifications as are suited to the particular uses contemplated.