Answers to frequently asked questions about Heurist can be found below. This information may be updated as the need arises.
Can I run Heurist without setting up my own server?
Yes. For those who do not have access to a free service through their university or national research infrastructure, we can provide a free service for small to medium databases on a University of Sydney server or various other servers around the world. See Installation.
Can I run Heurist on my own server?
Yes. You can install your own instance of Heurist on your own server or virtual server. Heurist is primarily intended to run on a Linux web server, and is easily installed on standard ISP virtual servers. We have used services costing as little as US$50/year from BuyVM.com with good results (this is not an endorsement, merely an observation – there are many other ISPs offering similar services). See Installation
Can I use Heurist on a stand-alone computer?
Although Heurist is primarily intended to run on a Linux web server, it can also run locally under Linux or Windows. However you will need some technical skills to get it running on a Windows machine. See Installation.
What does it cost to use Heurist?
Heurist is a free, open-source database platform. The Heurist team can provide you with additional assistance in setting-up and developing your database (including setting up a dedicated commercial virtual server in the Cloud). Please contact us to discuss your requirements and costs.
Is Heurist free and open source; where can I get the source code?
Yes. Heurist is a free, open-source database platform. Source-code is available to download from the Heurist Google Code page. See Installation.
Who supports and maintains the development of Heurist?
Heurist is being developed and supported within Sydney University’s Faculty of Arts and Social Sciences and the University’s Research Portfolio, under the direction of Dr. Ian Johnson. See History & Background.
How did Heurist come to be developed; who is behind Heurist?
Heurist was originally designed by Dr Ian Johnson, founding director of Arts eResearch (formerly the Archaeological Computing Laboratory), University of Sydney, and developed there from September 2005 to December 2013. Ongoing development and maintenance is supported by the Faculty of Arts and Social Sciences and the University’s Research Portfolio. See History & Background.
What happens if Heurist ceases to be developed and/or supported?
Heurist databases are standard MySQL databases (the world’s #1 Open Source SQL server database) and all share exactly the same structure of tables and fields. The data in any Heurist database can be accessed and exported in a coherent and usable form independent of Heurist through standard MySQL queries which can be used in a variety of database tools such as the MySQL command line, MySQL Workbench, PHPMyAdmin, or practically any programming language. Files and images are stored unmodified on disk in a separate directory for each database, and can be accessed with normal disk management tools (e.g. SSH Secure Shell or WinSCP) for downloading from a Unix server to a desktop. See Sustainability.
Can I get my data out of Heurist if I want to use another system?
Yes. Heurist can export selected data or the entire database in various formats including CSV files, XML files, and custom output formats generated through a built-in templating system. Heurist data can also be accessed directly through MySQL using a standard set of documented SQL queries, as all Heurist databases are standard MySQL databases (the world’s #1 Open Source SQL server database) and all share exactly the same structure of tables and fields. The data in any Heurist database can thus be accessed and exported in a coherent and usable form independent of Heurist through standard MySQL queries which can be used in a variety of database tools such as the MySQL command line, MySQL Workbench, PHPMyAdmin, or practically any programming language. Files and images are stored unmodified on disk in a separate directory for each database, and can be accessed with normal disk management tools (e.g. SSH Secure Shell or WinSCP) for downloading from a Unix server to a desktop. See Sustainability.
Can I transfer my database easily from one server/service to another?
Heurist has an option to dump the entire contents of the database to a Zip file which you can download. You will then need someone familiar with the Unix command line to reload this data on the new server following the instructions below. The process is not complicated and should take only a few minutes. The full manual process, which some people will prefer, is as follows: 1. Transfer the database: Dump the contents of the MySQL database to an SQL textfile using the mysqldump utility (include the –routines parameter), then create the new database on the new server and reload the dumped file (create database hdb_xxxx; use hdb_xxxx; source xxxx.sql;) 2. Transfer the file data: copy the contents of the database folder (identified by the name of the database) from the HEURIST_FILESTORE folder on the source machine to the equivalent folder on the target machine. Then set the ownership to nobody:nobody. See Sustainability.
How sustainable is a Heurist database?
Heurist databases are actually MySQL databases with identical, well-documented structure. MySQL is the world’s number one SQL server database and is likely to be readable for many years. Heurist databases will remain readable and the data will be usable, without the need for Heurist, as long as MySQL is around and backwards compatible. See Sustainability.
Can I access data in a Heurist database without using Heurist?
Since Heurist databases are MySQL databases with identical, well-documented structure, the data in any Heurist database can be accessed and exported in a coherent and usable form independent of Heurist, through a standard set of documented SQL queries.
Is my data secure in Heurist?
There are two aspects to this question – reliability and privacy. First, we have been running the current version of Heurist for nearly 5 years, and to the best of our knowledge neither we nor any other user has had any significant data loss in normal use. In terms of privacy, we do our best to ensure that Heurist data is protected from unauthorised access with encrypted passwords and levels of access permission, but as we all know even the biggest IT companies get hacked. We do not have their security resources, so we do not recommend using Heurist for personal confidential details, but we have the advantage that most Humanities research data is of such little commercial value that there is very little likelihood of theft.
Who else is using Heurist?
Please visit our listing of individual Projects for a list of some of the people and groups currently using Heurist. Heurist is also involved in two national Infrastructure Projects, the Humanities Networked Infrastructure, for which it provides database-on-demand functionality, and the Federated Archaeological Information Management System, for which it provides data modelling and automated creation of the tablet app design, as well as ingest and analysis of data and export to the tDAR repository.
Is Heurist suitable for large and/or complex databases?
Heurist is designed for small to medium databases, up to a few hundred thousand records. However, it can handle databases with many different entity types and linkages between entity types – unlike standard SQL databases, it is largely immune to the complexity of the data model. It is thus ideal for setting up complex databases with limited resources.
What types of data can Heurist store?
Heurist supports a large range of data types, including: text, term lists (categories), numeric, date/temporal, media files (image, pdf, audio, video, xml, kml etc.), geospatial data (points, lines, polygons, KML, shapefiles, tiled image files), record pointer and relationship marker fields. Record pointer and relationship marker fields are a powerful methodology for building relationships without the complexity normally associated with modelling multiple entities and their relationships. Media files in a File field may be stored directly within the database or may be referenced through an external URL, such as Youtube or Vimeo.
Should I put my bibliographic data in Heurist?
Although Heurist has the capability of storing bibliographic data in a clean normalised structure, we recommend the use of a specialised bibliographic system such as Zotero. No generic database can compete with a system built from the ground up for the peculiarities of bibliographic data and specialised functions such as in-text citation. Zotero libraries can be stored on the desktop and synched with a web copy. Heurist will synchronise a database with one or more Zotero web bibliographies, allowing one to reference full bibliographic data from entities within the Heurist database while using an optimised bibliographic tool to manage the bibliographic data.
Can I use Heurist for bookmarking websites?
Heurist started life as a web bookmarking tool and provides a bookmarklet tool to capture the URL, title and any selected text from a web page, and create a thumbnail of the page. It can also scan a web page or html file (including your browser’s bookmarks html file), selectively import hyperlinks and create records for them.
Will Heurist store photographs, scans and other media files?
Heurist supports a wide variety of file formats. Common image formats are automatically thumbnailed and can be viewed at different resolutions and annotated. PDF files are also viewable directly within the interface. Audio and video may be stored in the database, but are better stored on Youtube or Vimeo and referenced through a File field (see next FAQ question). Other file types (e.g. Word documents, html files, Flash, spreadsheets etc.) may also be stored and will generally open in the appropriate player or program if installed. In all cases, a file may be referenced by a URL on another website and provided it is accessible for download it will operate exactly the same as if it were stored in the database.
Will Heurist store videos; what video formats does it support?
Although Heurist will store and play video files, there are many problems associated with playback of video files, not least their size and variety of formats. We strongly recommend storing them in a specialised video services such as Vimeo or Youtube, which will stream them in optimised form to whatever device is in use. The video file is then referenced through its URL stored in a File field (File fields allow local upload but also accept a remote URL). The video will generally then play as an embedded video as though it was stored and streamed directly from the database.
Does Heurist support uncertain and fuzzy dates common in historical data and different calendars?
Yes. Heurist date fields can record dates in years (for many archaeological and historical applications), in day-month-year format for most contemporary dates, or as uncertain dates which have a range of probability associated with them. Dates can be described in terms of terminus post/ante quem, probable date range, and likelihood of position within the range of uncertainty. C14 dates with statistical deviations are also supported. Heurist can also store and convert dates in several common date formats including Julian, Gregorian, Islamic, Hindu and Buddhist calendars.
Do I need assistance from a programmer to set up Heurist?
If you are using a Heurist service, you should not need technical assistance. You can quickly build, develop and maintain your databases through point and click interactions with a web browser interface. The only area where some technical skill is required (at the level of simple html) is in writing specialised output formats in the report formatter function, or if you wish to use XSL transformations to generate output from Heurist’s XML stream. Most users will never need a programmer.
Would it take me less time to set up a database in Heurist than in an alternative system?
Not just less time, but much less time. Heurist builds complete functioning databases in a few seconds, and then allows you to rapidly add to and extend these databases through a web browser interface. The process of building databases is further accelerated by Heurist’s ability to selectively import new entity types, fields and term lists from exemplar databases and templates set up by the Heurist team, and from any other database registered with the central Heurist clearinghouse. Once you are familiar with the system, most projects can be set up in an hour or so and even the most complex and challenging projects will not require more than a few days.
Can I change my mind about database structure after I’ve entered data?
Absolutely. One of the great strengths of Heurist is that you can modify and enhance the structure of your data at any time, while maintaining the integrity of the data you have already entered. This allows you to start your database quickly, and over time building up its complexity and sophistication as your research needs evolve, without ever having to migrate existing data. See Define Database Structure.
Can I import an existing database structure to get started?
Heurist’s database creation function gives a choice of initial database templates. You can then selectively import the structure from any database which has been registered with the central Heurist index, including carefully structured exemplars and documented templates for specific domains. The record types, field types and term lists imported from other sources can then be edited as required to fit your specific needs.
How easy is it to add new entity types to my database?
Very easy. When a new database is created, it automatically inherits a default set of widely used record types from the chosen template. Additional record types can be imported from any database registered with the central Heurist index. In addition to editing these imported record types, you can click the Add Record Type button and define a new one by filling in information on two popup forms. See Define Database Structure.
How do I link entity types together to show relationships?
The simplest and most powerful ways of linking records are record pointer fields and relationship marker fields. These provide contextualised relatonships – prompts directly within the data entry form for a record – which establish a simple one-way relationship with another record (record pointer) or a complex two way relationship with another record (relationship marker). Like any other field type, both record pointer and relationship marker fields can be made optional or required, single or repeating, and can also be constrained to specific target record types. Relationship markers have additional attributes – the type of relationship in each direction, the time range for the relationship (if applicable) and other information about the relationship, such as notes and bibliographic justification of the relationship. See Connect Data.
Can I reorder and rename fields in a form without affecting existing data?
Yes. Within the database itself, all record types, fields and terms are identified by numerical codes. Field names, prompts, terms etc. are simply human-readable and do not affect the underlying data. They can thus be changed at will. In fact, the structure is so robust that a record can be changed to a different type and then changed back without loss of data, even if some of the fields did not exist in the intermediate record type. Fields can also be reordered at will.
How can I verify the integrity of my database?
Select Analyse > Data consistency from the Database menu. This analyses the data in the database to ensure that it conforms to the data structure and reports any discrepancies (e.g. missing required values, repeat values for a single value field, pointers which point at the wrong record type or a non-existant record etc.). Such discrepancies are uncommon but can occur over time if a lot of structural change has been carried out. Where discrepancies occur, the verification function provides ways of navigating to the records in question to fix them.
How do I go about importing data from an existing database into Heurist?
Heurist supports many import formats, including: CSV, KML, Bibliographic, Media. Once you have exported the original data using the relevant format, you can choose the corresponding import feature or wizard within Heurist to import the data. See Import Data.
My photographs are on an existing system can I reference them through the web?
Yes, using the File field type. File fields can reference an external URL or an uploaded file in the database.
Can I bulk upload multimedia items to my Heurist database?
Yes. Heurist’s In-Situ Bulk File Import tool let’s you quickly import thousands of files (such as images) See Import Data.
Can I import spatial data such as KML files?
Heurist supports the import of KML (Keyhole Markup Language) data, a file format used to display geographic data in an Earth browser such as Google Earth.
Can I publish data from Heurist to a website?
How do I publish data from Heurist to WordPress or another CMS?
(See the Roehampton Campus Project page for an example.)
Can I customise the Heurist interface to add Heurist search functions to my website?
This capability is available in Heurist version 4.
How Easy is it to write a custom report format for outputting data from Heurist?
Quite easy if you are prepared to poke around a write a little simple html.
Can I style custom reports to match my web site?
Yes, custom reports can reference styles, internal CSS or external CSS files just like any other html page.
Can I write custom PHP to extend the Smarty report function?
Yes, but only if you have backend access to the server (if you’re writing PHP you probably do).
Analysis and Visualisation
Does Heurist support complex search criteria?
Yes. An assisted search dialog can help you build complex, editable search strings. A complete search syntax is provided. See Search/Filter.
Does Heurist have a facility for faceted searches?
This capability will be available in Heurist version 4 (due in 2015). A preview is available on request.
How can I analyse the data in my Heurist database?
Heurist provides basic cross-tabulation of up to three fields, with counts and percentages. Cross-tabulation can be applied to the results of any search, allowing segmentation of the data.
The data can be visualised through a network diagram (applied to any search result set) as well as a map + timeline view. The network diagram view will export the nodes, edges and counts to a Gephi GEFX file for further analysis.
Data can easily be exported from Heurist in a variety of text formats including comma and tab separated files, and custom output formats can be developed with the report formatter. Custom analysis functions can also be added as tabs in the right-hand panel of the interface with PHP programming.
Can Heurist visualise the structure (schema) of my database?
Yes, in both list form and as a spring graph structure diagram.
Can Heurist visualise the structure of my data?
Yes, as a network diagram with popup information on the data fields and relationships connecting entities.
Can I export the data from my Heurist database to an analysis package?
Data can easily be exported from Heurist in a variety of text formt including comma and tab separated files and GEPHI GEFX, and custom output formats can be developed with the report formatter.
Mapping and Timelines
What do I have to do to create a map in Heurist?
Enter spatial data through a Geospatial field. Heurit provides a digitising function in the data edit form, but can also import spatial data as KML or WKT field data.
Can I use my own scanned Maps as base layers in a Heurist map?
Yes, they should be uploaded as a File in either a simple map layer record or served from a tile server and referenced in a tiled map image record. These map layers can then be referenced in a map document which will be used to draw the map and legend (functionality due late 2014 – current mapping can use single map images and KML only).
What do I have to do to create timeline in Heurist?
Simply recording one or a pair of date fields in a record will cause it to appear on the timeline when it is part of the result set being visualised. Records with both spatial data and date information will appear on both the map and the timeline and may b filtered from the map when they do not appear on the currently visible section of the timeline.
Sharing and Security
What is the most secure setup for running Heurist?
Heurist can be run – in order of security – on a standard web server (http), an encrypted web server (https), on a server accessible only on a private network (eg. VPN or LAN) or locally on a desktop/laptop machine. The service we provide for free is an http service, although we also run an https service which we can make available. Running Heurist locally is probably the least interesting path, because that’s really just turning it into a desktop system.
So the best option, if you need increased security, is to run your own server (which has the advantage that you control the jurisdiction in which it operates) on a private network (which your IT support ought to be able to set up for you); if not, we could help you set it up on a commercial web server such as Amazon AWS and lock it down to access by specific IP addresses.
While anything on the web can be hacked (as many of the major commercial players have found to their cost, despite their technical resources), the upside is that no-one is that interested in hacking academic Humanities information – it’s just not exciting enough or challenging enough to the hacker mentality, and there isn’t much of a buck to be made out of it
Heurist has a permissions model similar to Unix; that is, there are users and groups: records can be locked down to specific user visibility, group visibility or open to public view. Records are only ever editable by the owner, be that a specific user or a user group (users can belong to multiple groups).
Can I publish parts of my data to the web without exposing my whole database?
Yes, all records have a visibility setting. For them to be visible to people who are not logged in to the database, set them to Public.
Can I share my data selectively with other users of my Heurist database?
Yes. Database owners can set up workgroups and add members to workgroups, and control what record types can be viewed by its various members. Administrators of a workgroup have similar rights but only for their workgroup.
How do I make parts of my databases available to colleagues without giving them access to all my data?
Yes, all records have a visibility setting. You can set some of your records as visible to logged in users, others as owned by a group and visible to members of that group, and yet others to be owned by yourself and visible only to yourself. In all cases, only the group or individual owning the records can modify them.
How do I publish my database structure so others can use it?
Simply register your database with the Heurist central index (in Designer View select Database > Registration) and other Heurist users will be able to browse your data structure and download the parts they need. This will not give them any access whatsoever to your data.
How do I publish my custom reports so that others can use them?
At the present time they need to be exported (function in the report formatter), copied to the other user’s database, and then imported. We will make this a more practical function when time permits. You could also just exchange the contents via a text file and email, although the report may need some editing at the receiving end depending on how well the database structures correspond.
Standards and Terms
Does Heurist conform to international standards?
Heurist does not enforce any specific metadata standards, but may be configured to take advantage of any appropriate standard. Once a standard has been configured within the system, that database structure maybe copied by other databases through the Import structure function.
Can I import existing sets of terms into my Heurist database?
There is an Import button in the term tree editing function which will read a text file containing terms, one to a line, and optionally extended labels and standard alpha/numeric codes, if available.
Does Heurist allow me to export data as in as Linked open data?
Heurist does not have specific support for linked open data, but both the report formatter and the XSLT transform function can be configured to generate linked open data. Please contact the Heurist team for more information.
Can Heurist cope with non-roman and right to left alphabets such as Arabic?
Heurist can handle any alphabet which is supported by Unicode, including right-to-left alphabets
Can the Heurist interface be converted to languages other than English?
Key phrases in the Heurist interface are recorded in an internationalisation file, and have been translated into French and Russian. Additional languages can be supported by editing this file and adding the appropriate translations. Interface language is set in the Profile menu.
If I need help with setting up Heurist, is there someone who can help me?
The Heurist team is very happy to configure simple databases at no cost. Simple databases include those with a small number of entity types or standard entity types which we can import easily from existing work. Please contact us and provide a description of the entities you wish to record, including any specific attributes and terms which you wish to record. Please provide lists of terms as a spreadsheet or text file. For more complicated databases, specific workflows or customised reports, we will be happy to quote.
What is the support hours for the Heurist team?
The Heurist team generally operates within the Australian Eastern time zone, however at times members of the team are travelling internationally, extending support hours. Email requests for support will generally be answered within a few hours, if not immediately, from 8 AM to late at night AEST. Worst-case turnaround will be 1 to 2 days. We are also available for Skype or telephone conversation, but please send email first to arrange a suitable time.
Where can I get help with my Heurist database?
In the first instance we recommend contacting the Heurist support team who will be able to assist or who can put you in touch with Heurist users in your area.
If I need development of a custom function who can I turn to?
The Heurist support team is able to assist with the development of custom functions, or can provide technical advice to other programmers on how to go about developing and integrating appropriate functions. We welcome participation in the hearest open source project, and will be pleased to assist those who wish to become part of it.