WWWalker's Data Integration

See also Skills | Portals | Groupware | Finance Management | Hadoop | Hive

We have information management and software engineering qualifications so can use these to perform data integration effectively and efficiently.

We have worked on large data cleaning projects:

• importing 1 million patent records using Web services and text files
• importing and reformatting 182 HTML pages and related images
• importing financial records from Netbank into Web portal MySQL database
• converting TransXchange to Google Transit Feed Specification for public transport timetable mashups on Google Transit

We optimised the importation of the 1 million records from 58 hours to 2 hours by data cleaning using awk and bash scripts before importing into MySQL. We also ran batch imports using PHP.

We use various UNIX tools:

• wget
• awk
• sed
• bash
• python
• PHP
• MySQL
• C

and opensource tools like transxchange2gtfs

Modifying Bulk PDFs

We have modified 80 PDFs by rotating them 90 degrees left.

eBay Seller Consulting

We can remotely setup eBay auction items for sellers:

• finding images
• finding missing data in descriptions
• setting up PayPal
• setting up shipping and postage
• setup of and management of online stores

This is a huge exercise in logistics and supply chain management!

Big Data

In 2012 or later, I plan to write scripts for Hadoop and Hive to process big data for business intelligence using Amazon Web Services or similar.

Contact Us

Contact us if you need your data integrated, imported or cleaned.

WWWalker Intro