Blog

Blog Archives - 5 Record(s)

Year: '2015' - Month: '3'

20
March 2015

Gavin Pickin

ColdFusion JSON serialization failure - Unable to serialize binary data to JSON

CFML Language, CFML Server, Migrating to Railo, Techie Gotchas

Recently we finally migrated some big legacy websites off of some ColdFusion 8 servers. Yes, 8, we were still running one last server, and to be honest the only things running on it were BIG higher traffic websites, that we had not got the time to migrate. We have been migrating from Windows to Linux, and CF8 to CF9 and Railo / Lucee. Baby steps, but with our legacy code not having test coverage, it became a long painful process. You would think moving from ColdFusion 8 to 9 is a breeze, and for the most part it is, but I ran into a strange issue, which I am sharing today.

We have a series of CFCs (thankfully this code uses some) which we use for ajax requests. In these CFCs we have functions, with access = remote, and return format = JSON. We return the query, and ColdFusion serializes the data for us, and thats really all there is.

When we lit the site up on our dev server in CF9, we noticed some of our ajax calls were failing. The error returned was this:

JSON serialization failure. Unable to serialize binary data to JSON.

I googled a while, and found a few people having the issue, but not many solutions… hence, I am writing this now. I finally tracked it down to an issue with one field in my data. My CFCs were returning address data for my mapping system, and the code was erring out on the HTMLDATA field, which was a concatenation of the street number, street name, city state zip, phone and website url. So how was this concatenation of strings, becoming binary, and freaking out the JSON serializer?

The issue was, some of the fields, like Zip and Street number in the MySQL db were not strings. The concat function was actually joining strings and ints together, and since they were not all strings, it thought it should be binary (but only in CF9, CF8 has no issue with this approach).

The solution?

Cast those ints into strings, using the MySQL function cast( integer-field as char).
Remember this concat was long, has lots of fields, all over my cfc, I wish there was an easier way.
Finally it clicked, duh, I can just cast the concat… so only use 1 cast, which has to be quicker too, and easier to update.

So my code when from looking like this

concat(streetnumber, streetnumbermodifier, ' ', streetdirection, ' ', streetname, ' ', streetpostdirection, ' ', streetpostdirection, ' ', unit) as theaddress,

#to 

cast( concat(streetnumber, streetnumbermodifier, ' ', streetdirection, ' ', streetname, ' ', streetpostdirection, ' ', streetpostdirection, ' ', unit) as char) as theaddress,

 

Now, ColdFusion sees the concatenated information as a string, and can easily serialize that into JSON.

I have come across this myself a couple of times, if you are reading this, I hope this helps you too.
Keep an eye out for more migration tips, as I am making a big push to shut that server down, ASAP.

Thanks for reading
 

16
March 2015

Gavin Pickin

Speaking at Dev.Objective() x 2 - Cordova Hooks and Testable JavaScript

cfObjective, Conferences, Cordova / Phonegap, Dev.Objective, Javascript, Lucee, Mobile Development, Unit Testing

Dev.Objective() is the 10th year of cf.Objective() rebranded to describe the 5 track professional developer conference more accurately ( more info here).  Still hosting a large number of ColdFusion topics, and a lot of the normal speakers, the content again is simple amazing. Technology is changing and so is this conference, and I am happy to say, that not one, but two of my sessions have been picked for this years Dev.Objective() this May, 12-15th in Bloomington, Mn. 
As usual, my work fuels my blog, and my ideas for sessions, and this year is no different. Lately I have been focusing on Javascript with my Mobile Development, and I have come a long way, compiling knowledge as I go. I think both of these topics are great combinations of all the solid content out there, all in one place to get someone up and running quickly. I was surprised to get both accepted, more work than I was expecting, but I like a challenge, and work well under the gun, so its going to be fun knocking these out. If you are thinking about what conference to attend, you can’t do much better than this one.

Here is a quick look into my sessions.

First Session: How Do I Write Testable JavaScript?
Everyone who wasn't writing JavaScript, probably is now. Atwood's Law: any application that can be written in JavaScript, will eventually be written in JavaScript. That’s great, but how do we test it. In ColdFusion we have CFCs, most languages have classes... but JavaScript doesn't have classes (yet). So how do I write unit tests, what units are there, and how do I make my code look like that.
JavaScript is a flexible language, and with great flexibility comes great complexity and responsibility. Take your JavaScript spaghetti and make it unit testable.
Read more: http://www.devobjective.com/sessions/how-do-i-write-testable-javascript/

Second Session: Getting your Hooks into Cordova - Workflows and Build Helpers
Working in the web world, we have workflows and tools we are used to. We edit, save f5 reload to test our work as we go. Working with mobile, even Hybrid Mobile, is a very different workflow. Cordova lets us build hybrid apps but how do you test them, how do you interface with the Cordova build process, how do you debug. This session helps you go from "hello world" demo, to a professionally built app.
Read more: http://www.devobjective.com/sessions/getting-your-hooks-into-cordova-workflows-and-build-helpers/

One more big piece of news I think, is LUCEE the new CFML Engine, a fork of Railo with all the power of the community behind it has just announced they will be Platinum Sponsors. I cannot wait to talk to them and get more information on the future of our language.

Tickets are now on sale, go pick one up now.
Dev.Objective() Sessions
Dev.Objective() Speakers

Hope to see you there.
 

11
March 2015

Gavin Pickin

Windows Store Metro App issues with jQuery Ember Angular - Solved

Angular, Javascript, jQuery, Mobile Development, Techie Gotchas

During my recent adventures with cross platform app development, I have come across a few issues,.. and solved them along the way. I had an old issue just reappear. Since the Windows Platform gives you Javascript API access to some lower level features, there are some additional security you have to deal with, one of those includes Adding Dynamic Content. This of course is a key part of jQuery, Ember, Angular and Knockout, so this by default will make developing apps for the windows store and windows devices troublesome.

Text from Image: Javascript runtime error: Unable to add dynamic content. A script attempted to inject dynamic content, or elements previously modified dynamically, that might be unsafe. For example, using the innerHTML property to add script or malformed HTML will generate this exception. Use the toStaticHTML method to filter dynamic content, or explicitly create elements and attributes with a method such as createElement. For more information...

I had run into this previously, and you will find a lot of information how jQuery 1.x was facing this issue, and appendTo released a fix. When jQuery 2.0 came out, they pulled the fix, and jQuery 2.0 worked. I had my app running great, and then just recently it stopped again, so I had to find another solution. All of these information refers to 2013, and 2014, and it seems like this issue reemerged in 2015.

Solution Options?

There are ways you should wrap all of your dangerous calls with a special function, like this.

MSApp.execUnsafeLocalFunction(function () {
     $(‘#myDiv’).innerHTML = “I’m dangerous”;
}

 

Of course, you would have to find all such occurrences in your code, and in the source files, which is troublesome to say the least.
This is also not very cross platform friendly, unless you shim that as well.

I found no great out of the box solution, until I ran across one of my new favorite companies. MS OpenTech. Microsofts Open Source company, which I used previously when looking for a cross platform storage options, and how they solved that with a WebSql wrapper of SqLite. What they did in this case, is build a special shim, which works for all the common networks, and only runs in the case it is needed. The solution is Winstore-JSCompat
The github repo is here: https://github.com/MSOpenTech/winstore-jscompat

As long as you include this file early in your page, it will run, and shim all the dangerous functions so your code works, without all the headaches.
I dropped it in, and hey presto, that annoying message is gone, and my App is now running in my Windows 8 simulator. Well, it was, until I hit some WebSql syntax that it didn’t like. I’ll save that for another post.

Again, well done MS OpenTech.

05
March 2015

Gavin Pickin

Did you turn it off and on again? Troubleshooting Gotchas with Cordova and WebSQL

Chit Chat, Mobile Development, Techie Gotchas, WebSQL

If you have not seen IT Crowd, you are missing out on some strange but funny geek culture... and one of the highlights of that show is the line, "Did you turn it off and on again". Sometimes with troubleshooting, the simple things can throw you off, and that happened with me recently, so I wanted to share my frustration. This might not be the most informative post, but hopefully it helps someone, so they don't shoot themselves, like I was tempted to do so.

Of course, this might start a flame war, but those mac users know, turn it off and on again usually refers to a Windows machine, and that's what I had to use, again. I am working on this cross platform app, so the fun came when it was time to make sure everything was still working on windows 8. Short answer was NO NO NO NO NO.

First things first, it is a cordova app, using Visual Studio Express 2013 (although I need to get the Free Community Full Featured Version soon) and the Cordova CLI. With cross platform, I put all my files in dropbox, so it could sync behind the scenes, and life would be good. I spin up windows, go get a drink, make that 2, come back and check dropbox was synced, do a fresh build for windows

cordova build windows

Open the project in Visual Studio, and then run to my Surface (ok its not my surface, its the one I stole from my wife's college bag when she wasn't looking). Debugging shows an error... great, so no login screen, straight into an error. I'm feeling good, lets see what it is.

Connection ID not set

Now, this error was pulling up inside the plugin files itself. Strange. Searching on the google, its a very weird bug, and nothing to do with cordova, or the msopentech websql plugin. I am completely lost. I turn off everything talking to the database, and no error, turn on any database talking code, except the connection, and it throws the same error. Since the error discussed the connection, I throw the first DB call after the db.openDatabase into a setTimeout and make it wait for 10 seconds, overkill, to make sure the connection is made. No problem there, until the function runs, and then the same error.

In a frustrated growl, I decided I would removed and re-add the plugin, and remore and re-add the platform in cordova, to force the latest versions of the files.

Build and run, and boom - another error I solved earlier. The cordova platform for windows included an expired cert.
yourCordovaProject\platforms\windows\CordovaApp_TemporaryKey.pfx
There is a stack overflow question here http://stackoverflow.com/questions/26874331/multi-device-hybrid-app-fails-to-build-the-certificate-specified-has-expired and links to MS Open Tech Site where you can download a new file to replace the old one. https://msopentech.com/blog/2014/11/11/cordova-certificate-issue-were-working-on-it/
Download and replace that, and build, and run... and wait for the error.

GONE!!!

What? How did my App get messed up?
I have no clue, all I know is, sometimes you need to uninstall and reinstall, and it magically fixes everything. Maybe it was a new version, which I doubt, since I was checking version numbers. Maybe dropbox sync failed and a file got out of whack.

I don't know, but I was happy... its working right.

Looking at my login screen, I login, and boom. The whole app dies. Not a debugging error, with the dreaded break or continue. No... just boom dead. I comment out pieces, try to step through code. After battling the last problem, I say screw it... I'm shutting down Visual Studio, I'm going to reboot the machine.

I got kick some walls, flip some tables, attack a few villages, have a drink, and when windows comes back up... I cordova build windows, open Visual Studio, run and ta-daa, I see the real error. No more app crashing without control, I get the normal break / continue debug issue, and now, I can figure out whats wrong.

I assume Visual Studio's project wasn't refreshed after adding and removing the platform, or something, but turning it off, and on again fixed it.
Now, after my day is done, I'm back where I started, with the error I was planning to fix for the day.

Moral of the story.
If something doesn't make sense, uninstall reinstall. If it still makes no sense. Turn it off and on again.

Happy debugging everyone :)

 

03
March 2015

Gavin Pickin

Loading or Dumping Large Amounts of Data into WebSql - Options with Performance Numbers

Javascript, Mobile Development, WebSQL

Working with databases, there is going to be a time when you need to dump large amounts of data out of, or into your database. Dumping data out, has always been pretty easy, but what about inserts? We’ll look at a couple of solutions, and a couple of issues, and see what seems to be the best solution.

I mentioned in an earlier post “Using WebSQL in my Mobile App - Lessons Learned - The hard way”  on WebSQL that although that standard covers a pretty good amount of SQL, but not everything is supported on each implementation. In Safari the following syntax works, but it will not in chrome

'INSERT INTO myTable ( ID ) values ( ? ), ( ? )’, [ null, null ]

 

In chrome, as far as I know, each insert, can only insert one record. I am going to look at the different ways to do a bulk insert, using the single insert syntax

'INSERT INTO myTable ( ID ) values ( ? )’, [ null ]

 

Scenario 1 - 500+ Transactions - 500+ Queries

Normally, I wrap my inserts with a transaction, so my code will look something like this.

create: function( companyData ){
    app.db.transaction(function (tx) {
   
     var thesql = 'INSERT INTO company ' +
                    '( ' +
                    'ID,' +
                    'company,' +
                    'updatedAt,' +
                    ') VALUES (' +
                    'null, ' +
                    '?,' +
                    '?)';
     tx.executeSql(thesql,
                    [
                        companyData.company,
                        companyData.updatedAt,
                    ],
                    function( tx, data ){ console.log('done'); },
                    function( tx, error ){
                         console.log( 'error inserting trainings in local db');
                         console.log( error );
                    }
                   );
          });
   
     }


This works well for simple inserts, but when you are bulk inserting, the setting up and tearing down the transaction, adds up. Especially when you are doing 500+ transactions.

Running scenario 1: 15000 - 15500 milliseconds

 

Scenario 2 - 1 Transaction 500+ Queries

Lets setup a special batch function, and we’ll pass in the array to loop over inside the transaction, executing 1 transaction, but 500+ queries.

createBatch: function( companyData ){
   
  app.db.transaction(function (tx) {
          var thesql = 'INSERT INTO company ' +
               ( ' +
               'ID,' +
               'company,' +
               'updatedAt,' +
               ') VALUES (' +
               'null, ' +
               '?,' +
'?)';
          for (var i = 0; i < companyData.length; i++){
               tx.executeSql(thesql,
                    [
                    companyData[i].company,
                    companyData[i].updatedAt,
                    ],
                    app.companyDAO.genericSqlResult,
                    app.companyDAO.genericSqlError
               );
          }
     });
}

 

The only difference, other than wrapping the tx.executeSql inside the for loop is the fact that I removed the anon functions, and used named functions. Why did I do that? First, JS Hint told me to… which is pretty smart really. Having 500+ anon functions is probably not smart, having 1 function, and calling it, from each of the loops is a big win for efficiency.

Running scenario 2: 2200 - 2500 milliseconds

 

Scenario 3 - 1 Transaction - 1 Queries - 500+ Values

Now, assuming we are in a WebSQL environment that supports the Safari syntax, where 1 query can have multiple inserts, we should see even faster times. I wonder how much faster. I adjust my code to match this syntax

'INSERT INTO myTable ( ID ) values ( ? ), ( ? )’, [ null, null ]

 

To do this, we need to loop over the number of values, and then add all of the parameters into one big array. This is how I did this.

createBatch2: function( companyData ){
   
     console.log('query starting');
     app.db.transaction(function (tx) {
          var thesql = 'INSERT INTO company ' +
               '( ' +
               'ID,' +
               'company,' +
               'updatedAt,' +
               ') VALUES ';
          for (var i = 0; i < 500; i++){
               thesql = thesql +
                    '(' +
                    'null, ' +
                    '?,' +
                    '?)';
                    
                    // add comma after each set of values except last
                    if (i < 499 ){
                         thesql = thesql + ',';
                     }
                }
                //Add all values into parameter array
                var paramArray = [];
                for ( i = 0; i < 500; i++){
                    paramArray.push( companyData[i].companyID );
                    paramArray.push( companyData[i].company );
                    paramArray.push( companyData[i].parentID );
                    paramArray.push( companyData[i].updatedAt );
                    paramArray.push( companyData[i].datetimestampticks );
                    paramArray.push( localTimeStamp );
                }
                //execute the query
                tx.executeSql(thesql,
                    paramArray,
                    app.companyDAO.genericSqlResult,
                    app.companyDAO.genericSqlError
               );
     });
}

 

Run the query, and now I get an error. Great, what did I do wrong. Here is the error.

{"code":5,"message":"could not prepare statement (1 too many terms in compound SELECT)"}

 

After some googling, I find out that SQL Lite has a max of 500 values statements by default… so to get it to run, I just switch out companyData.length with the number 500. Now when I run it, I get a result.

Summary of Results

Running scenario 1: 15000 - 15500 milliseconds - 561 records
Running scenario 2: 2200 - 2500 milliseconds - 561 records
Running scenario 3: 90 - 100 milliseconds - 500 records

Of course scenario 3 only inserts 500 records, but the numbers are obviously much more performant with Scenario 3, with 500, or 561 records.

In summary, if you are doing bulk imports, you need to know your environment, because the fast method, is only available on some implementations. If you are using scenario 3, you will need to batch your queries into 500 value chunks, or override the limits as specified at SQLite.org 

If you are not sure if you can use multiple values inside a single query, batching your queries into the same transaction is still much better than using a normal create DAO function, so take the extra time to implement a batchCreate method too.

I hope you find this interesting, and the code helps you out when working with your WebSQL Database, tackling Large Bulk Data Imports and Inserts.
Now I know what method is best, and what I'll be using it in the future.

Blog Search