Posts

My days as a BigData engineer grew darker in the past few days. Now almost all critical data that our aggregations, partners and customers want to access is located in a bucket on Amazon’s S3. Since that gets horribly expensive for the vast amounts of data we create on a daily basis, we lean towards not keeping much more than a month of active objects in the bucket, to meet our SLA’s regarding backup and recovery, we keep them for about a year in Glacier afterwards. ...

So it was late at night and I was hacking away at my touch-controls for my UnicornHat. I was trying to set the brightness in nine steps between 0.1 and 0.9. When all of a sudden, this happened: >>> foo=0.1 >>> foo+=0.1 >>> print(foo) 0.2 >>> foo+=0.1 >>> print(foo) 0.30000000000000004 So according to my internal senses this looked not very precise. But according to the almighty Stackoverflow and one of my beloved coworkers who can indeed speak ASM fluently, this is in fact very precise (for a float). Since a float is merely a value, lets say, close by the one you wished it to be. ...

I just hit a point in an app I am currently developing, where I found myself generating queries from the resultset of another query. Because, what can possibly go wrong, right? $selectors = array(); $query_phase2 = "SHOW KEYS FROM `".$row['table']."` WHERE Key_name = 'PRIMARY'"; $query_phase3 = "select `".$row['field']."` from `".$row['table']."` where "; if($result = $GLOBALS['dbconn']->query($query_phase2)) { while ($row = $result->fetch_assoc()) { if(array_key_exists($row['Column_name'], $data)) { $qextender = "`".$row['Column_name']."` = '".$data[$row['Column_name']]."'"; array_push($selectors,$qextender); } } $query_phase3 .= implode(" and ",$selectors); if($result = $GLOBALS['dbconn']->query($query_phase3)) { if($result->num_rows > 1) { $return = array(); while($row = $result->fetch_array()) { array_push($return,$row[0]); } } elseif ( $result->num_rows == 1) { $return = $result->fetch_array(); return $return[0]; } else { return ""; } } }else { return array('danger' => 'query failed('.$GLOBALS['dbconn']->errno.'): '.$GLOBALS['dbconn']->error);}

I just stumbled across an enormous outcry about another attack by the “LizardSquad” against DBG’s (former SOE) Games that were allegedly the outcome of a few angry posts on Twitter by DBG’s CEO John Smedley. (Storylink) Along with the usual “fixit fixit fixit” there was also a lot of guesses thrown out about the size of the genitals of a 14-year old fin. What startled me though is the sheer lack of knowledge when it comes to the gamers themselves. So please, let me help and explain how this magical and dangerous DDoS stuff works. ...

I just viewed a very nice #31c3 talk about a well known issue with certain functions within two very widely used perl-modules (DBI & CGI). When I heard the arguments brought to the presenter from the audience within the Q&A, I thought to myself, well, they’re right. All of these attack vectors are in theory something that could have been prevented by just flatten arguments (as you always should). Then again, this makes for a really, really messy programming style that does not come easy if you were programming any other programming language prior to perl. Lucky me, I started programming with perl. I think within the third script I used the first own functions and within my fourth script I realised that it is probably best for everyone to start all your functions this way: ...

I recently commenced the third installation of JAVA on one of my Debian-Boxes and couldn’t remember any of the exact commands, had to google again, ended up with 8 open tabs all saying something different and so wrote this small bash script to set up all necessary alternatives with “update-alternatives” automatically. Please note that there has to be either icedtea or openjdk installed to make all parts of this script work. ...

I recently got certified with cloudera (CCAH) and found myself the only one in class who really uses Hadoop. I often stumble upon people trying to sell Big Data appliances to brand owners “to store and monitor their social network XXX” where XXX may be replaced by any word that tickles or frightens brand owners. My personal opinion is, if you are a brand owner or not, doesn’t matter, data matters. And only data generated and maintained within your own company matters. If you want to throw away lots and lots of money, than you can start storing data which is freely accessible to anyone. ...

Just finished a script, that enables me or anyone else who cares to send tweets directly to a IRC-Channel. The script is powered by the best IRC-Client ever made, perl, and Net::Twitter::Lite. Update [2023-08-18]: Nobody should use twitter or perl or irc any more ;) Update [2013-06-15]: upgrade to Net::Twitter::Lite::WithAPIv1_1 Update [2013-01-12]: second update today added the setting “_count” and some cleanups with vars and setup Update [2013-01-12]: drastically reduced the amout of requests to the API P.S.: you can also follow me on twitter, but beware that almost half of my tweets will be neither about programming nor in english ;-). Follow @matschundbrei, if you don’t care! ...

I was just writing a small script for automatizing something via a REST-API and I had to manually add an authorization to all my GET-Requests, which really annoyed me. (it was not Realm-sensitive, so I couldn’t set up auth in the UA itself). So I wrote this small sub I would like to share: #getrequests shorthand sub getReq { my ( $url, $username, $password ) = @_; my $req = HTTP::Request->new( GET => $url ); if ( $username && $password ) { $req->authorization_basic( $username, $password ); } my $res = $ua->request($req); if ( $res->is_success ) { return $res; } else { warn $res->status_line; return undef; } } All you need to call a HTTP-Base-Auth’d site now is: ...

Today I do not have very much to do at work. I am not moaning about it, I would just like to point out that my actual team has a very strict “don’t change anything that works before a holiday or the weekend”-policy, this is a very good policy, but it leads to personal downtimes on fridays and the days before holidays for obvious reasons that I like to fill with personal development. So I began learning something about screen and the modelines which are used in it. I found a comment from someone which was pointing out his zshrc which one should look up for a funcion namend “screen_title”. I immediatly searched the web for his name and zshrc, and the only thing that came up was his blog. Unfortunately I was sucked in by his latest post, pointing out a presentation on slideshare and completly forgot about my intention visiting this site. From this presentation forked some talks about databases and bigdata and complexity that I would really like to share. Let’s start at the beginning. My interest for Hadoop as a technology was mainly generated by a stong powersource, namely the catalog of videos over at Cloudera and a whitepaper by Greenplum which was pointed out by Dmitriy Ryaboy in his video about Pig at twitter. ...