Ilya Goldin, Ph.D.
Human-Centered Data Science
ilyagoldin@gmail.com
RECENT HIGHLIGHTS
US Patent US10027740B2 on technology for adaptive learning
Presented at National Academy of Medicine SIG on Education of the Health Care and Science Workforce
Goldin, I., Galyardt, A., published in Journal of Educational Data Mining
EDUCATION
Post-Ph.D. fellowship, 2013. Carnegie Mellon University, Human-Computer Interaction
Ph.D., 2011; M.S., 2007. University of Pittsburgh, Artificial Intelligence
B.S., Magna Cum Laude, 1999. Stevenson University, Liberal Arts + Information Systems
SKILLS
Data Science and Statistics: machine learning, data mining, natural language processing, information retrieval, visualization, R, generalized linear mixed effects, Bayesian and graphical models
Research: experiment design, qualitative and quantitative data collection and analysis
Educational Technology: adaptive learning, personalized learning, learning sciences
AWARDS
MacArthur Foundation / Educational Testing Service Edmund W. Gordon Fellow, 2013-2015
PostPIER Postdoctoral Training Grant, Institute of Educational Sciences, 2011-2013
Outstanding Paper Presentation Award, University of Pittsburgh Grad Expo, 2006, 2001
Faculty of Arts and Sciences Fellowship, University of Pittsburgh, 1999
President’s Award, Stevenson University, 1999
Outstanding Scholar Award, Stevenson University, 1999
First place, Computer Scholarship Competition, Stevenson University, 1996
EXPERIENCE
Principal Data Scientist, Phenom People, 2019-present
Using AI and Data Science to help 1B people find the right job.
Director, Data Science, 2U, Inc., 2015-2019
Developed data-driven algorithms for digital learning: to describe fundamental limitations on discovery of domain models; to align noisy log files to expected event schedules; to define a computational model of course content and learning outcomes. Primarily working in R, these inventions led to internal products and scholarly publications.
Provided organizational guidance on data science, learning science and educational technology. Worked with product management to design new feature agenda, and to articulate market positioning and differentiation.
Research Scientist, Center for Digital Data, Analytics & Adaptive Learning, Pearson, Inc., 2013-2015
Developed data-driven algorithms for digital learning: ways to probabilistically estimate student mastery of a skill, to discover prerequisite skill relationships, to flag learning activities for editorial attention, and others. Primarily working in R, these inventions led to internal prototypes, patent applications and scholarly publications.
Planned and supervised field-testing of Pearson technologies with hundreds of K-12 students in Utah and Michigan, and derived insights from data that drove product revisions.
Leveraged academic background and relationships to bring outside expertise to Pearson projects, to support colleagues' professional development, and to enhance Pearson's position as a thought leaders in educational technology.
Post-doctoral Fellow, Carnegie Mellon University, 2011-2013
Studied the effect of hints on learning by creating novel statistical models for data from tutoring software, and by designing and conducting in-school studies. Presented research results at academic conferences.
Data Administrator, Magee-Womens Research Institute and Foundation, 2010-2011
Processed hundreds of gigabytes of data for biomedical informatics tasks using custom and third-party software and scripting in a Unix environment.
Research Assistant, JustSystems Evans Research, Inc., 2006-2009
Designed, developed and evaluated software and interfaces in machine learning, recommender systems, semantic modeling, visualization, natural language processing and ontologies.
Research & Development Intern, McCabe & Associates, 1997-1998
Worked on software quality assurance, including unit, regression and function testing.
FULL-LENGTH REFEREED PUBLICATIONS
Goldin, I., Galyardt, A. (2018) Most of the time, it works every time: Limitations in using learning curve analysis to refine domain models. Journal of Educational Data Mining, Vol. 10(2), pp. 55-92
Galyardt, A., Goldin, I. (2015). Move your lamp post: Recent data reflects learner knowledge better than older data. Journal of Educational Data Mining, Vol. 7(2), pp. 83-108
Galyardt, A., Goldin, I. (2015). Evaluating simplicial mixtures of Markov chains for modeling student metacognitive strategies. Quantitative Psychology Research, Vol. 89, pp. 377–393
Goldin, I., Pinkus, R.L., Ashley, K. (2015) Validity and reliability of an instrument for assessing case analyses in Bioengineering Ethics education. Science and Engineering Ethics, Vol. 21(3), pp. 789–807
Goldin, I., Renken, M., Galyardt, A., Litkowski, E. (2014) Individual differences in identifying sources of science knowledge. Proceedings of 9th European Conference On Technology Enhanced Learning
Almond, R. G., Goldin, I., Guo, Y., Wang, N. (2014). Vertical and stationary scales for progress maps. Proceedings of 7th International Conference on Educational Data Mining
Goldin, I., Carlson, R. (2013) Learner differences and hint content. 16th International Conference on Artificial Intelligence and Education
Goldin, I., Ashley, K. (2012) Eliciting formative assessment in peer review. Journal of Writing Research, 4(2), pp. 203-237.
Goldin, I., Koedinger, K., Aleven, V. (2012) Learner differences in hint processing. Proceedings of the 5th International Conference on Educational Data Mining. [Finalist for Best Paper Award]
Ashley, K., Goldin, I. (2011) Toward AI-enhanced computer-supported peer review in legal education. Proceedings of the 24th International Conference on Legal Knowledge and Information Systems
Goldin, I., Ashley, K. (2011) Peering inside peer review with Bayesian models. Proceedings of the 15th International Conference on Artificial Intelligence and Education
Goldin, I., Ashley, K. (2010) Eliciting informative feedback in peer review: importance of problem-specific scaffolding. Proceedings of the 10th International Conference on Intelligent Tutoring Systems
Goldin, I., Ashley, K., Pinkus, R.L. (2006) Assessing case analyses in Bioengineering Ethics education: Reliability and training. Proceedings of the 8th International Conference on Engineering Education
Goldin, I., Ashley, K., Pinkus, R.L. (2001) Introducing PETE: computer support for teaching ethics. Proceedings of the 8th International Conference on Artificial Intelligence & Law
ADDITIONAL PUBLICATIONS
Goldin, I., Narciss, S., Foltz, P., Bauer, M. (2017). New Directions in Formative Feedback in Interactive Learning Environments. International Journal of Artificial Intelligence in Education.
Goldin, I., Pavlik Jr, P., Ritter, S. (2016). Discovering domain models in learning curve data. Design Recommendations for Intelligent Tutoring Systems: Domain Modeling (Vol 4). US Army Research Laboratory
Goldin, I., Galyardt, A. (2015). Convergent validity of a student model: Recent-Performance Factors Analysis. Proceedings of 8th International Conference on Educational Data Mining (EDM)
Goldin, I., Galyardt, A. (2015). On Feasibility of Using Learning Curve Analysis to Refine Domain Models. Meeting of the Advisory Board of the Generalized Intelligent Framework for Tutoring
Goldin, I., Galyardt, A. (2015). Viz-R: Using Recency to Improve Student and Domain Models. Proceedings of 2nd ACM Conference on Learning at Scale
Forsgren Velasquez, N., Goldin, I., Martin, T., Maughan, J. (2014). Learning aid use patterns and their impact on exam performance in online developmental mathematics. Proceedings of 7th International Conference on Educational Data Mining (EDM)
Galyardt, A., Goldin, I. (2014). Recent-Performance Factors Analysis. Proceedings of 7th International Conference on Educational Data Mining (EDM)
Scheines, R., Silver, E., Goldin, I. (2014). Discovering prerequisite relationships among knowledge components. Proceedings of 7th International Conference on Educational Data Mining (EDM)
Goldin, I., McLaren, B. (2014). Towards instrumenting collaborative learning and assessment in the digital ocean. Proceedings of the Workshop on Social, Motivational and Affective Dimensions of Learning through Social Interaction, at the 11th International Conference of the Learning Sciences (ICLS)
Goldin, I., McLaren, B. (2014). Towards instrumenting collaborative learning and assessment in the digital ocean. Proceedings of 3rd Workshop on Intelligent Support for Learning in Groups, at the 12th International Conference on Intelligent Tutoring Systems (ITS)
Renken, M., Litkowski, E., Goldin, I., Oliver, M. (2014). Students’ Recognition of the Value of Scientific Process When Evaluating Science Explanations. Presented at the American Psychological Association Annual Conference (APA)
Goldin, I. (2013) Towards Supporting Students and Instructors with Models of Peer Assessment. Invited symposium. 78th Annual Meeting of the Psychometric Society (IMPS)
Galyardt, A., Goldin, I. (2013). Modeling Student Metacognitive Strategies in an Intelligent Tutoring System. Presented at 78th Annual Meeting of the Psychometric Society (IMPS)
Goldin, I., Koedinger, K., Aleven, V. (2013) Hints: You Can't Have Just One. 6th International Conference on Educational Data Mining (EDM)
Goldin, I. (2013) Examining Peer Reviewer Calibration with Bayesian Models. It's About Time: Addressing the Many Challenges of Analyzing Multi-Scale Temporal Data. Workshop at Alpine Rendez-Vous
Goldin, I., Carlson, R. (2013) Learner Differences and Hint Content. Data Analysis and Interpretation for Learning Environments (DAILE). Workshop at Alpine Rendez-Vous
Goldin, I., Ashley, K., Schunn, C. (2012) Redesigning Educational Peer Review Interactions Using Computer Tools: An Introduction. Journal of Writing Research, 4(2), 111-119.
Goldin, I. (2012) Accounting for Peer Reviewer Bias with Bayesian Models. Proceedings of the Workshop on Intelligent Support for Learning Groups at the 11th International Conference on Intelligent Tutoring Systems
Ashley, K., Goldin, I. (2012) Computer-Supported Peer Review in a Law School Context. U. of Pittsburgh Legal Studies Research Paper No. 2012-24.
Goldin, I (2011) A focus on content: The use of rubrics in peer review to guide students and instructors. PhD dissertation, University of Pittsburgh.
Goldin, I., Ashley, K., Pinkus, R.L. (2006) Teaching case analysis through framing: Prospects for an ITS in an ill-defined domain. Proceedings of the Workshop on Intelligent Tutoring Systems for Ill-Defined Domains at the 8th International Conference on Intelligent Tutoring Systems
Goldin, I., Chapman, W.W. Learning to detect negation with ‘not’ in medical texts. Proceedings of the Workshop on Text Analysis and Search for Bioinformatics at the 26th International ACM SIGIR Conference
PATENT APPLICATIONS
Mahe Bayireddi, Hari Bayireddy, Sivanand Akella, Suresh Babu Devineni, S.S.S. Venkateswara Rao Alapati, Shivaramakrishna Nyshadham, Ilya Goldin. "Knowledge engine using machine learning and predictive modeling for optimizing recruitment management systems". U.S. Patent Application US16/843,805 (International PCT US20-27329).
Gonzalez-Brenes, J., Goldin, I.M., Larusson, J.A., Behrens, J., McTavish, T., Rho, Y.J., Anderson, J.M., Kukartsev, G.A. "Content Database Generation". U.S. Patent Grant US 10,713,225 B2
Gonzalez-Brenes, J., Goldin, I.M., Larusson, J.A., Behrens, J., McTavish, T. "System and method for increasing data transmission rates through a content distribution network with customized aggregations". U.S. Patent Grant US 10,027,740 B2
Scheines, R.; Silver, E.; Goldin, I. "Discovering Prerequisite Relationships among Knowledge Components." U.S. Patent Application 62/188,184
Evans, D.A.; Bennett, J.; Hull, D.; Cheng, H.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Anchoring Expressions Based on an Ontological Model of Semantic Information.” U.S. Patent Application 20080294426
Hull, D.; Evans, D.A.; Bennett, J.; Cheng, H.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Performing Semantic Update and Replace Operations.” U.S. Patent Application 20080294425
Evans, D.A.; Bennett, J.; Hull, D.; Cheng, H.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Performing Semantically Informed Text Operations.” U.S. Patent Application 20080295013
Cheng, H.; Evans, D.A.; Bennett, J.; Hull, D.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Performing a Semantically Informed Merge Operation.” U.S. Patent Application 20080294427
Qu, Y.; Evans, D.A.; Goldin, I. “Method and Apparatus for the Automated Construction of Models of Activities from Textual Descriptions of the Activities.” U.S. Patent Application 20080294398
RESEARCH GRANTS
Community Care Behavioral Health Organization. Analysis of Disengagement in Mental Health Provider Networks. PI, 2011, $12,000.
Provost's Advisory Council on Instructional Excellence, University of Pittsburgh. A Peer Review-Based Student Model for Ill-Defined Problem-Solving. Co-author with K. Ashley, 2009, $25,000.
PUBLIC SOFTWARE
Curtis, S.M. (author), Goldin, I. (contributor). (2012) mcmcplots: visualization of Markov-Chain Monte Carlo output for Bayesian modeling. http://cran.r-project.org/web/packages/mcmcplots
Goldin, I. (2012) ProfHelp and ProfHelp-ID: multilevel Bayesian models of learner differences with hint processing.
Goldin, I. (2012) Peer reviewer bias: multilevel Bayesian models of bias in peer assessment and bias-instructor score relationship.
Goldin, I. (2011) Rubrics in peer review: multilevel Bayesian models of multidimensional rubrics in peer review.
Goldin, I. (2010) Comrade: web-based peer review software. Used in 4 studies by 200 students and 3 instructors.
Goldin, I. (2001) Professional Ethics Tutoring Environment (PETE): web-based ethics tutoring software.
INVITED TALKS
Data-Driven Curriculum Design for Medical Education. National Academy of Medicine SIG on Education of the Health Care and Science Workforce (2018)
Applications of Data Science in Education. Ontario Institute for Studies in Education, University of Toronto (2016)
Applications of Data Science in Education. Centre for Teaching, Learning and Technology, University of British Columbia (2016)
Domain Modeling Using Learning Curve Analysis. Learning Analytics Summer Institute-local, New York University (2016)
On Formative Assessment in Interactive Learning Environments. Learning Sciences Institute, Arizona State University (2013)
Peering into Peer Review with Bayesian Models. Institute for Language, Cognition and Computation, University of Edinburgh, UK (2013)
Peering into Peer Review with Bayesian Models. Institute of Educational Technology, The Open University, UK (2013)
Towards Effective Feedback for All: Learner Differences in Hint Processing. Learning Analytics Seminar Series, Teachers College, Columbia University (2013)
Towards Effective Feedback for All: Learner Differences in Hint Processing. Artificial Intelligence Forum, Intelligent Systems Program, University of Pittsburgh (2012)
ACADEMIC SERVICE
Guest Co-editor
International Journal of Artificial Intelligence in Education, Special Issue: Formative Feedback in Interactive Learning Environments (2017), with Foltz, P., Narciss, S., Bauer, M.
Journal of Writing Research, Special issue: Redesigning Peer Review Interactions Using Computer Tools (2012), with Ashley, K., Schunn, C.
Program Committee Member
International Conference on Intelligent User Interfaces (2016)
International Conference on Artificial Intelligence in Education (2015-2018)
International Conference on Educational Data Mining (2013-2019)
Artificial Intelligence and Feedback, a workshop at International Joint Conferences on Artificial Intelligence (2015)
Data Mining for Educational Assessment and Feedback (ASSESS), a workshop at KDD 2014
International Conference on Creating, Connecting and Collaborating through Computing (2011)
Journal Reviewer
International Journal of Artificial Intelligence in Education (2016, 2015, 2013)
IEEE Transactions on Learning Technologies (2018, 2013)
Journal of Educational Data Mining (2018, 2011)
Technology, Instruction, Cognition and Learning (2013)
Journal of Educational Psychology (2012)
Conference Reviewer
ACM CHI Conference on Human Factors in Computing Systems (2014)
Society for Research on Educational Effectiveness (Spring 2013)
International Conference on Artificial Intelligence in Education (2011)
International Conference on Educational Data Mining (2012-14, 2010)
International Conference on Computers and Education (2009)
Conference of Florida Artificial Intelligence Research Society (2009, 2008)
Grant Application Reviewer
National Science Foundation, USA (2013)
Economic and Social Research Council, UK (2008)
Chair
Workshops, International Conference on Educational Data Mining (2016)
Interactive Events track, International Conference on Artificial Intelligence in Education (2015), with Olga Santos
Intelligent Support for Learning in Groups, a workshop at International Conference on Intelligent Tutoring Systems (2014) and International Conference on Artificial Intelligence in Education (2015), with Roberto Maltinez-Maldonado, Erin Walker, Jihie Kim, and Cindy Hmelo-Silver
Formative Feedback in Interactive Learning Environments, a workshop at International Conference on Artificial Intelligence and Education (2013), with Taylor Martin, Ryan Baker, Vincent Aleven and Tiffany Barnes
Crosspollination: Multidisciplinary Approaches to Human Cognition and Behavior, a workshop at inter-Science of Learning Centers (iSLC) Conference (2012), with Amy Ogan and So-One Hwang
Computer-Supported Peer Review in Education: Synergies with Intelligent Tutoring Systems, a workshop at International Conference on Intelligent Tutoring Systems (2010)
Committee Member
Doctoral dissertation committee, University of Southern California, EdD candidate Erika Maldonado (2018)
Internal Volunteering
Liaison to Executive Committee of the Pittsburgh Science of Learning Center (PSLC) from PSLC postdoctoral fellows (2012-13)
MENTORING
Laura Malkiewich, intern, 2U, Inc.
Yetian Chen, intern, Pearson Education, with Jose Gonzalez-Brenes
Ryan Carlson, Ph.D. student, Carnegie Mellon University, with Kenneth Koedinger and Carolyn Rose
LearnLab Summer School: Educational Data Mining track, Pittsburgh Science of Learning Center (2012, 2013)
Timothy Hirsh, intern, JustSystems Evans Research, Inc.