Heurist’s research-driven data management system allows any confident researcher or data manager to design, create, manage, analyse and publish their own richly-structured database(s) within hours, through a simple web interface, without need of programmers or consultants.
Use Sydney University’s free service (click the Try It button) to build a new database in minutes and a complete application in hours. Design and build incrementally; existing data are not affected. Nor are you locked in; migrate later to a personal or institutional server, or access data directly via SQL.
Whatever your field of study – History, Archaeology, Anthropology, Cultural Research, Linguistics, Media Studies etc. – Heurist handles everything from text, tags and categories, through networks of rich relationships and offset annotations, to images, video, spatial data and dating (including non-western systems and dating uncertainty).
Get results: Sophisticated searches (including faceted and rule-based) can be saved, organised and rendered as layered interactive maps, timelines, network visualisations, cross-tabulations, lists, custom reports, XML and XSLT transforms, all of which can easily be printed or embedded (live) in a website.
Designed by researchers, for researchers, Heurist reduces complex underlying decisions to simple, logical choices.
Heurist’s intuitive web-based interface is uniquely tailored to the data needs of Humanities scholars, rather than to business or scientific data. Any confident researcher should be able to build a fully-functional database in minutes, and develop a complex database tailored to their needs in hours or days, all without programming.
Heurist’s ability to import, and then adapt, a library of useful structures developed by other users is both a great time-saver and a powerful force for standardisation without compulsion. Simple forms with helpful tips provide for the easy creation of new entity types and relationships, addition and modification of fields, addition of new terms, data verification, bulk edits, data import and data export.
PROBLEM: You need: a proper database, several entity types, relationships between entity types, full control of your information, access from multiple locations, selective sharing with colleagues, compatibility with standards, sustainable and archivable data, free and open source. You need to be able to: change database structure at a moment’s notice, import existing data, reference external data, have assistants safely enter data, publish selectively to a website, map and timeline the contents, network graph the content, do simple (cross)tabulation, and export data for analysis and visualisation. And: you don’t have a technical team, you don’t have much/any funding, and you need it today!
SOLUTION: Heurist resolves the problem that eResearchers have typically faced in managing their complex research data coming from different sources and in different forms (e.g. bibliographies, research notes, images, map data etc.).
As an alternative to either building a dedicated database (time-consuming and expensive) or knocking together an ad-hoc solution using whatever tools are to hand (creating a ‘silo’ environment), Heurist offers researchers an integrated, free and Open Source single web-based tool to handle all their data, allowing them to build their own, complex databases without investing in technical expertise.
Any researcher can now rapidly create a simple database in minutes and a complex one in hours; a database with a host of advanced features not normally available without special programming, such as the ability to: reuse data models created by other users; publish information to the web; access information from anywhere; share data (selectively) with colleagues; or visualise data through maps, timelines and network charts.
Simple forms provide helpful tips that allow for the easy creation of new entity types and relationships, addition and modification of fields, addition of new terms, data verification, bulk edits, data import and data export.
And Heurist incorporates features such as repeatable fields, simple creation and use of relationships (record linking), automatic mapping and timelines, on-the-fly record structure change, and sharing of pre-configured database schemas through a central clearing-house.
To get started, you can use the free service hosted at the University of Sydney Data Centre (heurist.sydney.edu.au) or one of the other listed services, or set up Heurist on your own server or cloud service in a couple of hours. Databases are easily migrated between servers.
Heurist’s set of powerful, robust and adaptable features extends its application to a wide range of scenarios and provides scope for multiple applicabilities. From a single system, Heurist users have the tools and flexibility to create a wide-range of solutions, only restricted by their imagination. Current applications include: a text and media-rich public encyclopedia of historical content; linked annotation and publication of literary works; artwork collections; medieval menus; field data collection via Android tablets; site survey data; complex stratified excavation data; content-rich life histories and connections; event records; bibliographic, administrative and job-tracking databases.
The following shows a sub-set of current and planned uses:
A text and media-rich public encyclopaedia of historical content for a city, with maps and timelines. | A visual representation of a city’s historic crimes and criminals. | A record of ballads, pamphlets, tunes and people. | Information on buildings, people and events on campus for use in an undergraduate course. | A bug tracking system. | A rental database. | A database of photographs from an archaeological survey of a country. | An interlinked set of descriptions for artists, paintings, collections, and villages for a country. | A history of people, sports, events across a single culture & city. | A visually-rich collection of medieval menus, recipes and ingredients.| Excavation phases, contexts, structures, finds, and type specimens. | Field survey and excavation data collected via tablets. | Records of educational service providers and their services. | A central repository of all kinds of information about cities with the ability to freely link and build as new data becomes available. | Interlinked life histories of WW1 soldiers, with rich photographic and documentary material, and timelines. | An archaeological record of Alpine sites. | A collection of photographs and videos of historic agricultural practices still predominately being performed in the Middle East, including a record of the types of activity, the landscape and the social context. | A collection of photographs synchronised with Filmmaker and Lightroom databases, making them available for download on-line. | A collection of relevant research information about the boundaries of Ancient Mesopotamia. | Index of historic maps of a major city. | Annotation and linking of historical texts. | Linked annotation and publication of literary work. | Artwork collections. | Medieval menus. | Field data collection via Android tablets. | Site survey data. | Complex stratified excavation data. | Content-rich life histories and connections. | Event records | Bibliographic, administrative and job-tracking databases.
Heurist’s complete database-design environment, incorporating sophisticated database facilities, lets you manage complex Humanities data and its structure, in an evolving manner. Get started immediately with something small and simple, and then modify and extend your database structure as you become more comfortable with it or your data model and needs evolve.
New databases take only a few minutes to set up, although an investment in downloading well-designed data structures (templates) and in fine-tuning these (e.g. eliminating unneeded fields, setting field labelling, field order, field widths, field requirements and repetition of values), will lead to a better user experience.
And you can continue to evolve your database design at any time without affecting your existing data.
Heurist’s import tools let you easily import and manage different data formats, including CSV, KML, Bibliographic, and Media. Heurist takes you step-by-step through the import of structured tabular data, from a range of sources: worksheets, CSV files, SQL tables, with entity matching on one or more key fields and merging of data.
Large volumes of images and other multimedia files can be quickly loaded in bulk and indexed. In addition, some forms of data can be synchronised; for example, Zotero bibliographies can be synchronised to connect objects of study with full bibliographic references.
Heurist supports the import of KML (Keyhole Markup Language) data, a file format used to display geographic data in an Earth browser such as Google Earth.
From simple text searches to faceted searches and rules for retrieving a network of connections, Heurist provides numerous options for searching and filtering databases. Search filters can be saved and re-used to analyse or publish data subsets.
A quick search tool assists you in building complex, editable filter expressions, to target particular record types, fields and values from controlled lists, without needing to know the required syntax. Refine your search criteria by:
- Entering partial strings to match against record titles, tags, keywords, special fields, or all fields.
- Using comparison operators (e.g. Greater Than) on numeric fields and dates, including relative to ‘today’.
- Searching for records of specified types or bookmarked by specific users across databases.
- Sorting records by text, date, popularity, rating, date of addition and modification, specific fields etc.
To build more complex search strings use the advanced ‘Assisted Search’ dialog. A complete search syntax is provided. Expand your search by building RulesSets, or refine the search through faceted searches.
The result of any search can be saved and reused, letting you create personal searches for grouping records for particular purposes (saved searches can be shared with any workgroups of which you are a member). Saved searches can also store your filter rules and ownership settings.
Analysis & Visualisation
Rule-based network expansion, faceted searching, cross-tabulation analysis, interactive maps, timelines and network graphs provide summarisation and visualisation of data patterns. Various output formats and user-defined custom reports can feed subsets to mapping, graphics, statistics and other visualisation packages.
Heurist has a rich-set of tools for analysing your data structure and data.
You can visualise the structure (schema) of your database, in both list form and as a network diagram. The network diagram provides an interactive (spring-loaded) visualisation of your database’s record types. Record types are shown as nodes, and the connections (pointer fields and relationships) as the links between nodes (edges). You can update links (relationships) directly via the diagram by simply drawing new connections. Data result-sets can also be visualised through a data network diagram.
A map & timeline view shows geospatial and timeline data stored for ‘map documents’ attached to one or more records.
Heurist provides basic cross-tabulation (of up to three fields, with counts and percentages) which can be applied to the result-set, allowing segmentation of the data.
Data can easily be exported from Heurist in a variety of text formats, including comma and tab separated files and GEPHI GEFX. Custom analysis functions can also be added to the interface with PHP programming.
Heurist provides an integrated workgroup environment for sharing and collaborating online with other users, through workgroups, sharing, tagging, saved filters, messaging, record-level ownership and publishing control (while maintaining your personal and private information.
In addition to been used by individual researchers who wish to create and use a database for their own projects, Heurist’s shared collaborative knowledge-space allows project members to share an overlapping set of records in a single database (securely available online) and communicate with one another around those records through a range of communication channels:
By setting up workgroups and adding members to workgroups, Heurist can be used for group projects, with tasks divided between those designing the database and those populating and searching it. Members can collaborate on research projects by sharing research data/functions with their colleagues (i.e. fellow members of the workgroup) through a range of collaboration tools. Members can display a subset of record types for specific groups of users, protected from modification or viewing by non-members.
Within shared database, researchers can ‘bookmark’ sets of records (your own records and other user records of relevance) to develop a personal subset of the shared database. You can search specifically within just bookmarked records, to quickly find commonly used records. Bookmarks are a way of separating private information about a record and shared information about a record; for any bookmarked record, you can attach information, either for your own personal use or for sharing:
- Add personal rich-text notes.
- Rate records (sortable in search results).
- Join group discussions per record.
- Leave comments for others to reply to.
- Add web links, web pages, images and geographic location to shared records.
- Attach personal password reminders to bookmarks for password protected resources.
Set workgroup tags (tag records with meaningful, multiple search keys) and share these among workgroup members. Tag creation includes multi-word tags, tag disambiguation, clickable lists of suggestions, instantly search by tags (including partial matching), and follow tags to related records. Manage tags through popularity, deletion, combining and renaming.
Add and share extended notes to any bookmarked record or engage in a discussion with other workgroup members.
Notify other users of records that might be of interest to them. Or attach email reminders to shared records to notify a user or workgroup about multiple records of interest at specified intervals (specific date, daily, weekly, monthly, yearly).
Record-level ownership and publishing control
Control workgroup members’ access and permissions, as well as access to published reports.
Use Heurist’s Smarty-based Report Builder to define custom reports to be printed or published in a variety of formats, including as live-feeds to a web, CMS or blogging site; Heurist even writes the HTML/JS code for you to paste into your web page editor, with no programming required. XML output can be transformed inline through stylesheets.
The Scheduled Reports function can be used to set up a call for writing reports to file (HTML and JS). Reports can be run on demand or based on a user-defined schedule.
Sustainability & Access
Sustainability of complex digital resources is a critical issue to anyone committing their research data to digital format. To this end, data is stored in a fixed format in the most widely used Open Source SQL database, allowing independent access through any programming tools into the future. Heurist also supports fully-documented generic SQL queries, XML feeds, XSL transforms, and creates JSON, XML & SQL archive packages.
Heurist databases have high sustainability—of the order of decades—for the following reasons:
1. Heurist is built on MySQL, the most widely used Open Source relational database server today (used by many major applications including WordPress, Drupal, Wikipedia, Facebook, Twitter, Tumblr, Flickr, YouTube among others).
2. All Heurist databases have an identical MySQL structure, which is internally documented with consistent field-naming conventions and a concise and informative comment section on every field.
3. The structure, data and logic of a database built in Heurist is built into the database itself, not the software; the structure, data and logic in a Heurist database are fully self-documenting. This contrasts with typical database applications where the database structure maps to the domain of the application and the application software is essential to using, if not interpreting, the content. Such applications are inherently unsustainable and probably have an unmaintained life of less than 5 years. The data in Heurist databases will be accessible and comprehensible/usable via standard SQL as long as MySQL databases are readable, which is likely to be a matter of decades.
One-click creation of sustainable archive package: Database owners/managers can export the complete contents of a database to an archive package, consisting of: a text file containing a MySQL database dump (the database rendered in SQL statements) which can be reloaded into any recent MySQL server (or other SQL server with a bit of work); a text file containing an XML rendering of the database content; any uploaded files (text, images, videos, spreadsheets etc.) in their original format; a textual description of the Heurist database structure.
Ancient History, Annotation, Annotation of Records, Archaeological Data, Archaeology, Art, Art History, Artifact, Artworks, Audio, Bibliographic Listing, Bibliographic Record, Bibliography, Biographical Data, Biography, Blog Writing, Bookmarklet, Central Index of Databases, Classical Studies, Collection, Connections, Cultural and Visual Studies, Data, Data Collection, Data File, Data Import, Data Modeling, Database, Digital and Physical Archival Collection, Digital Documents, Digital Humanities, Digitisation, Directed Relationships, End-user Structure Modification, Entity Extraction, Events, Excavation, Excavation Data, External Message Trigger, Faceted Search, Feeds, Field Data, File, Film Studies, Flexible Database Structure, Fuzzy Date, Fuzzy Date Value, Geographic Coordinates, Geographic Objects, Geospatial Data, Historical Data, Historical Map, History, History, Hyperlinks, Image Annotation, Image Management, Image Resizing, Images, Interactive Network Visualisation, Internal Messaging, Laboratory Data, Link, Literature, Manuscript, Map, Map Layers, Map with Timeline, Mapping, Media Studies, Metadata, Monument, Multimedia, Network, Network Visualization, Objects, Offset Text Annotation, Organisations, PDF, Persons, Photography, Place, Project, Prosopography, Relationship, Research, Rich Field Type, Rule-based Network Search, Simple and Advanced Search, Sites, Spatial Data, Spreadsheet, Structure Import, Structures, Survey Data, Tablet, Tablet Data Collection, Tagging, Task, Text, Time-stamped Relationship, Timeline, Tools, User-defined Report, Video , Web Publishing, Workgroup, Workgroup Ownership, Workgroup Permission, Workgroup-based Saved Search, XML, XML Output, XSLT, XSLT Inline Transform