More Proper Data Abstractions in .NET and Mono Applications

Hi Andy! 🙂 *waves*

In http://little.xmtp.net/blog/2005/10/21/proper-data-abstraction-in-net-and-mono-applications/, wow, almost a month ago, I talked about how to use Mono.Data.ProviderFactory to build data applications which are not tied to a particular ADO.NET data provider. I completely skipped using Parameters. I’d like to revisit it.

Remember that this is bad:

System.Data.SqlClient.SqlConnection conn = new System.Data.SqlClient.SqlConnection(connstring);
conn.Open();
System.Data.SqlClient.SqlCommand cmd = new System.Data.SqlClient.SqlCommand(cmdstring,conn);
System.Data.SqlClient.SqlDataReader reader = cmd.ExecuteReader(CommandBehavior.CloseConnection);

This is good:

IDbConnection conn = ProviderFactory.CreateConnectionFromConfig("CmDbStr");
conn.Open();
IDbCommand cmd = conn.CreateCommand();
cmd.CommandText = cmdstring;
IDataReader reader = cmd.ExecuteReader(CommandBehavior.CloseConnection);

This is a nice, simple example of using the IDataReader. What about IDbDataAdapter? The .NET Framework SDK Documentation says to create new DataAdapter instances like this:

SqlConnection conn = new SqlConnection(connection);
SqlDataAdapter adapter = new SqlDataAdapter();
adapter.SelectCommand = new SqlCommand(query, conn);
adapter.Fill(dataset);
return dataset;
...
OdbcConnection conn = new OdbcConnection(connection);
OdbcDataAdapter adapter = new OdbcDataAdapter();
adapter.SelectCommand = new OdbcCommand(query, conn);
adapter.Fill(dataset);
return dataset;

To do so would lock me into using a particular Provider for my connections.

Using Mono.Data to create a DataAdapter is definitely different, but it works well.

IDbConnection conn = ProviderFactory.CreateConnectionFromConfig("CmDbStr");
conn.Open();
IDbCommand selectCommand = conn.CreateCommand();
selectCommand.CommandText = selectCommandString;
IDbDataAdapter da = ProviderFactory.CreateDataAdapter(selectCommand);

Once the IDbDataAdapter is created, you can set the other commands.

da.InsertCommand = conn.CreateCommand();
da.UpdateCommand = conn.CreateCommand();
da.DeleteCommand = conn.CreateCommand();
da.InsertCommand.CommandText = insertCommandString;
da.UpdateCommand.CommandText = updateCommandString;
da.DeleteCommand.CommandText = deleteCommandString;

If you like using CommandBuilders such as SqlCommandBuilder or OdbcCommandBuilder, you are stuck here. Well, the enterprise developer in me says you should be using stored procedures for your CRUD, but the Agile goofball in me says that quick and dirty wins the race. Mono.Data doesn’t have any helpers for use with Command Builders. The problem is that in ADO.NET 1.0, Command Builders are second class citizens. There are no interfaces for it. SqlCommandBuilder inherits from System.ComponentModel.Component. ADO.NET 2.0 changes this, but Mono hasn’t implemented any provider factories yet. The provider model in ADO.NET 2 may not require it. I’ve yet to dive into ADO.NET 2 enough to say for sure.

If you notice above we filled a DataSet using da.Fill(dataset). It turns out IDbDataAdapter inherits from IDataAdapter. The Fill method is from IDataAdapter, so we can fill datasets with our ProviderFactory provided adapter with no problems.

The last piece of ADO.NET which I’ll address is parameters.

From the SDK Documentation again:

myDataAdapter.SelectCommand.Parameters.Add("@CategoryName", SqlDbType.VarChar, 80).Value = "toasters";
myDataAdapter.SelectCommand.Parameters.Add("@SerialNum", SqlDbType.Int).Value = 239;
myDataAdapter.Fill(myDataSet);

Wow, it looks like that might work on our IDbCommands too! But it turns out that Parameters is an IDataParameterCollection which doesn’t have an Add method. The above Add method is a member of the SqlDataParameterCollection. The quick and easy workaround it to case and use this add method.

IDbConnection conn = ProviderFactory.CreateConnectionFromConfig("CmDbStr");
conn.Open();
IDbCommand selectCommand = conn.CreateCommand();
selectCommand.CommandText = selectCommandString;
((SqlDataParameterCollection)selectCommand.Parameters).Add("@Param",SqlDbType.VarChar,80).Value="param val";
IDbDataAdapter da = ProviderFactory.CreateDataAdapter(selectCommand);

This works, but it immediately ties us to a particular provider again.

As mentioned in the last Mono.Data post, the DataTools class has some helpers for that.

IDbCommand idcSelect = conn.CreateCommand();
idcSelect.CommandText = sSelect;
DataTools.AddParameter(idcSelect, "@Username", DbType.String);
DataTools.AddParameter(idcSelect, "@Email", DbType.String);
((IDataParameter)idpcSelect["@Username"]).Value = Username;
((IDataParameter)idpcSelect["@Email"]).Value = emailaddres;

There are many excellent helper methods in the static class Mono.Data.DataTools. I’ve found reading the source to be the most useful way to figure out how to use them.

If you do look at the source for the AddParameter Method, you will see that it is damn simple:

IDataParameter param=Cmd.CreateParameter();
Cmd.Parameters.Add(param);
param.ParameterName=ParameterName;
param.Direction=Direction;
param.DbType=DbType;
return param;

This is a direct example on how to add parameters to your IDbCommand if you wanted direct access to that parameter without using the cast in the prior example.

Did I say that Parameters was the last thing I wanted to mention? I can’t stop there. This one is really important and it is something which I have not seen anywhere in ADO.NET documentation both online or in the SDK. I don’t have much use for paper books, but I would hope it is there.

I wouldn’t even know about this, or its importance had it not been for prior experience with Perl. Yes, that is right, Perl DBI! The DBI(3) perldoc/manpage (type perldoc DBI at your friendly shell prompt) is one of the best documents for good database interaction. Specifically it points out dos and don’ts for working with databases. It stresses the use of DBI->prepare and DBI->execute for successive SQL executions. While I would hope that use of DataSets and DataAdapters would do this for you, sometimes you just have to use a Command directly. When using a Command directly, repeatedly, it is important to not “prepare” every time. This happens behind the scenes. ADO.NET doesn’t directly have a “prepare” method, so it is not as easy to see the need to solve a problem which you don’t know you have until your application slows to a snail pace.

Above we created parameters and set their values in different steps. Here is how it may look when implemented poorly.

foreach ( Username in Usernames ) {
   IDbCommand idcUpdate = conn.CreateCommand();
   idcUpdate.CommandText = sSelect;
   DataTools.AddParameter(idcUpdate, "@Username", DbType.String);
   DataTools.AddParameter(idcUpdate, "@LastDate", DbType.String);
   ((IDataParameter)idcUpdate["@Username"]).Value = Username;
   ((IDataParameter)idcUpdate["@LasteDate"]).Value = DateTime.Now;
   idcUpdate.ExecuteNonQuery();
}

That is about as bad as it gets. A new command is created for each cycle of the loop. New Parameters, new statement preparation, etc…

It turns out that it is pretty damn simple to fix.

IDbCommand idcUpdate = conn.CreateCommand();
idcUpdate.CommandText = sSelect;
DataTools.AddParameter(idcUpdate, "@Username", DbType.String);
DataTools.AddParameter(idcUpdate, "@LastDate", DbType.String);
foreach ( Username in Usernames ) {
   ((IDataParameter)idcUpdate["@Username"]).Value = Username;
   ((IDataParameter)idcUpdate["@LasteDate"]).Value = DateTime.Now;
   idcUpdate.ExecuteNonQuery();
}

Here the command is created once, outside the loop. The statement is prepared with its parameters. The loop simply changes the parameter values and executes the query again. This is the equivalent of a Perl DBI prepare outside a loop with an execute inside the loop.

I hope this helps. I’d love to see some excellent Web and Database apps from the Open Source community which don’t tie users to a particular database. *cough*wordpress*cough*

Bye Andy!

QT Components for Ogg Vorbis for Win32

Anyone who was using ogg in QT/iTunes on Win32 has probably cried 1000 rivers full of tears since iTunes6/QT7 came out and broke the implementation from http://qtcomponents.sourceforge.net/. I know I have. I’m tearing up right now.

Last night while checking again to see if there was any new news on this front, I noticed that someone had released a Macho PPC version. It made me cry more. iTunes on Win32 is a second class citizen indeed. This is one place where banshee on Linux actually beats iTunes. For once, I decided to do something about it. I checked out qtoggvorbis from sourceforge and both ogg and vorbis from xiph. I converted that ancient VC6 project to my trusty Studio 2005 project and solution. I downloaded the QT SDK from Apple. I read. I read more. I read lots. I implemented (hopefully correctly) S64Compare. Apparently an implementation of this was in the QT6 libs but is gone from QT7? I’m not sure. I built. I resolved errors. I ignored 200+ compiler warnings. I changed some project paths. I added include paths. I added library paths. It finally built! I made sure the oggvorbis.qtx was in C:\Windows\System32\QuickTime. I started QuickTime. I openned an ogg. I crashed QuickTime.

I guess it won’t be a trivial port. But it has been fun so far. Next step is to connect the Studio 2005 debugger to the running QuickTime process and see if I can break it in my dll. I hope the qtx format doesn’t mangle this ability. It turns out these qtx files are just dlls with some extra apple stuff (I think things which apple normally places in resource forks, but win32 has no resource forks).

Pray for me to get something working so that we can all enjoy ogg in iTunes.

asp.net access is denied: … assembly could not be loaded

Wow, I didn’t think I’d run into conflicts with Index Server and ASP.NET, but I did.

http://support.microsoft.com/default.aspx?scid=kb;en-us;329065

There is really no need for Index Server to index the ASP.NET Temporary Files folder. Adding it to the Index Server exclude list is pretty easy. I’m just very surprised that this was the issue. I was guessing my error would be a CAS thing.

Oh well.

So if you get an ugly asp.net page error that says something about “Access is Denied: ‘YourClassNameHere’” and then some source junk to make you go down wrong paths, and eventually an “Assembly Load Trace: The following information can be helpful to determine why the assembly ‘YourAssemblyNameHere’ could not be loaded.“, and then some LOG: stuff, well then you probably need to follow the instructions in that there link.

Laymans guide to Managed vs. Unmanaged

Ok, so this GAIM converstation isn’t 100% acurate, but for explaining something to your non programmer friend, your mom, or you dog, it is good enough.

(09:38:29) yonoshe_steckler: what’s the diff between managed and unmanaged?
(09:38:47) sillyevar: managed is something that runs in a runtime like .NET or Java (or perl or python for that matter)
(09:39:10) sillyevar: unmanaged is something that runs right on the hardware/CPU of a machine, like compiled C,C++ and traditional programs
(09:40:02) yonoshe_steckler: but wait… i thought that java was a compiled language
(09:40:56) sillyevar: its compiled to java bytecode, not native bytecode(machine language)
(09:41:23) sillyevar: C# and .NET is the same way, it is compiled to .NET bytecode (called CIL, common intermediate langauge)
(09:41:42) sillyevar: then the JVM interpretes those bytecodes (or in the .net case, the .net runtime interprets them)
(09:41:51) yonoshe_steckler: oooooh
(09:41:53) yonoshe_steckler: ok
(09:42:20) sillyevar: that is why you have things like a JIT (just in time)… it supposedly does the JVM interpretation to machine code fast enough that you can’t tell the difference, and that may be true… BUT!!!
(09:43:38) sillyevar: both of these systems manage memory usage within the VM, and that means programs are allowed to write code with memory leaks (sometimes huge ones, think azaurus), and the VM will clean up the memory leaks, but that cleanup takes time. Sometimes it can take lots of time, like 50, 70, even 90 percent of the CPU time can be spend managing memory.
(09:44:37) yonoshe_steckler: yikes

dontstealmylaptop

free for the beta,
free for the first three months.
$4 per month or $40 per year
no theft retreival garanteed.

In the case of theft, we can work with the authorities for you for an extra fee.

Optional serial number tracking for insurance purposes.

probably want to know it is working… view a last communicated date.

making a laptop stolen causes all information to be saved. Normally all information is purged after 24hrs.

More bitching about MSSQL

Microsoft folks just LOVE to belittle MySQL. They call it fake and list all of the things which it didn’t used to be able to do(often citing facts which only apply to 3.x versions or 4.0 versions, but 5.0 is now the most recent). I’m no MySQL fan either. I prefer Postgresql.

However, this isn’t a database comparison post. This is a bitch about “the power that is” Microsoft SQL Server. I was going to refrain from bitching in hopes that 2005 would be better, but in this case it has the same limit! 8060 bytes per row (actually 8039, the Microsoft documentation LIES!) Yet according to the Launch event which I just attended, MSSQL has sold more licenses than Oracle and DB2 combined. I guess that is mostly because of all the MS Access upsizers and how cheap the MSSQL std. license is. It sure doesn’t seem to be because of any technical decisions. I could not find any information quickly googling for maximum bytes per row for Oracle.

I am starting to understand why the DBAs here are so anti-MSSQL and so pro-oracle. I hate oracle mostly because I find it cumbersome to use, I never worked with TOAD which seems to be the DBA’s preference, but I’m also not a DBA, so I never had the need. I’m a programmer whose footsteps into the realm of dba is intentionally minimal. I prefer the excellent readline history and online help of Postgresql’s psql. \h and \? are my best friends and so are ctrl-r and ctrl-p in emacs readline mode.

MySQL row size limits depend on the table type, for example 65,534 for MyISAM with BLOB and TEXT fields not counting toward that (they are stored elsewhere). I’m not sure what the max row size for InnoDB tables is.

I prefer the rediculously high size maximum for a row in Postgresql. The size is 1.6TB. That isn’t a typo on my part, but maybe the Postgresql FAQ is incorrect. Either way I would hate to see the speed of a table scan on a table with rows that size. 🙂 I knew I liked Postgresql, but this is just more fuel for me.

God help me show my fellow programmers the light. I want SQL standard EXCEPT. I want to use temporary tables in triggers, I want practically unlimited row lengths.

Php syntax (semantics) exam

I just finished the PHP Syntax Exam at http://www.blueshoes.org/en/developer/syntax_exam/ in 555 seconds and had 24 errors in 69 questions.

Do you think you can beat me? 🙂

Wow, whoever wrote this “exam” doesn’t even know the different between syntax and semantics. Syntax was not tested at all. Semantics was tested.

Honestly, I’m surprised I did as well as I did, given the horrible abomination of a language that is php.

At Microsoft Kickoff

Wow,

I have no idea how or why DETROIT is the #2 stop on the Microsoft Visual Studio 2005 Lauch tour. Maybe they think that after their big launch in San Fran, they need a list of crappy cities to work the kinks out of their tour. Don’t get me wrong, I love Detroit, I just didn’t think anyone else did.

Team System is the most integrated application I have ever seen. You can do everything from where it makes sense. Project managers can manage projects in Excel and still be working off the team server. It is really awesome. Of course, the pricetag is so insane I really don’t think very many organizations will adopt it.

The stuff about Team Foundation Server (or whatever the VSS replacement is) about concurrent versioning made me laugh. The presenter actually demoed an example of merge and I kept thinking, wow, I had this 10 years ago when I first used source control. It was called CVS.

So far I have seen a keynote and a Team System presentation. I’m looking forward to afternoon presentations about more products which I will probably never use.