עמותת המקור – פלאנט משתמשי תוכנה חופשית

אתר הכנס:
http://august.penguin.org.il
עמוד בוויקי

מרצים המעוניינים להעביר הרצאה בכנס מוזמנים לשלוח הצעה לוועד המקור board@hamakor.org.il או לוויקי בקטגוריה "תוכן הכנס".

Recently I had to install adaptec CLI management tool on an ubuntu 8.10 amd64 server. Despite of the aacraid driver present in kernel, it took me a lot of time to find the management tools that allow to see RAID status and manipulate it from command line. There are several names for the tools: afacli, aaccli, afaapps, afa-apps-snmp, arcconf, hrconf... oh my!
After I found the necessary tools (64bit arch) I packaged them into a tiny deb package adaptec-utils_0.0.1_amd64.deb, which can be downloaded from here: DELETED_AS_REDISTRIBUTION OF BINARIES_IS_PROHIBITED_BY_ADAPTEC

Also, I wrote a simple monitoring script (included in the deb package) that checks state of the Adaptec RAID system and sends out email if any of the published indicators is wrong (battery, failed disks, etc). The script can be downloaded from here: http://t11.mine.nu/adaptec-utils/adaptec_check.sh

Note that the script requires proprietary binary files to be installed. See the beginning of the script for the details.

‫קוראי הבלוג הקבועים כבר קראו שרציתי לעבור לצפון מפאת חסכון ועוד סיבות, ואכן עברתי לצפון. לאן בדיוק? בשלב זה, להוריי. מדוע? אה, טוב ששאלתם. כאן בצפת נפתח לאחרונה בית ספר לרפואה, מה שהצליח תוך זמן קצר להקפיץ את המחירים פה בעשרות (ולעיתים במאות) אחוזים. פעם היה אפשר לשכור בצפת דירה של 3 חדרים בגודל של [...]‬

A system I've been working on is based on the .NET framework. It seems that Mono and Apache on a web server is a pretty good combo, and I recommend everyone with a spare day or two to give it a try.

Anyway, I needed to run a blogging engine on that same server and have that blog support Hebrew and right-to-left languages. While there are several good blogging engines for .NET, not all would run on Mono and most doesn't do the RTL part good enough.
My first attempt was to use SubText, but getting rid of the SQL Server dependency and adding the RTL support became too much of a hassle. I picked SubText for it being notorious for its simplicity and elegance, but while I was ready to invest some customization time, it demanded too much of it.
My second attempt was BlogEngine.NET. What can I say - I fell for it almost immediately. The platform is versatile enough to support not only many RDBMS's, but it also allows storing the posts in XML files. Nonetheless, the translation to Hebrew wasn't complete and there was no RTL native theme for it. So I made one.

BlogEngine.NET is hosted on CodePlex and uses Mercurial for its source control. I had a fork of the project in no time, so I started making the required changes and created a new theme based on the default one. Since I don't want to maintain a fork of the project for eternity and since I want others to enjoy this new feature, I've sent a pull-request which was accepted yesterday. So starting from version 2.5.0.6, you can enjoy BlogEngine.NET for your Hebrew blog.

As always, it feels good to contribute back and to know that my code can be further enhanced by other developers.

Happy blogging.

In this post I describe what happens when a slave's Foreign Key setup is different from that of the master. I'm in particular interested in a setup where the slave has a subset of the master's foreign keys, or no foreign keys at all. I wish to observe whether integrity holds.

Making the changes

Which foreign keys do we have and how do we drop them? If you want to do this by hand, well, good luck! Fortunately, common_schema provides with quite a few handy views and routines to assist us. Consider viewing the existing foreign keys on sakila:

master> SELECT create_statement FROM common_schema.sql_foreign_keys WHERE TABLE_SCHEMA='sakila';
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| create_statement                                                                                                                                                                                |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ALTER TABLE `sakila`.`address` ADD CONSTRAINT `fk_address_city` FOREIGN KEY (`city_id`) REFERENCES `sakila`.`city` (`city_id`) ON DELETE RESTRICT ON UPDATE CASCADE                             |
| ALTER TABLE `sakila`.`city` ADD CONSTRAINT `fk_city_country` FOREIGN KEY (`country_id`) REFERENCES `sakila`.`country` (`country_id`) ON DELETE RESTRICT ON UPDATE CASCADE                       |
| ALTER TABLE `sakila`.`customer` ADD CONSTRAINT `fk_customer_address` FOREIGN KEY (`address_id`) REFERENCES `sakila`.`address` (`address_id`) ON DELETE RESTRICT ON UPDATE CASCADE               |
| ALTER TABLE `sakila`.`customer` ADD CONSTRAINT `fk_customer_store` FOREIGN KEY (`store_id`) REFERENCES `sakila`.`store` (`store_id`) ON DELETE RESTRICT ON UPDATE CASCADE                       |
| ALTER TABLE `sakila`.`film` ADD CONSTRAINT `fk_film_language` FOREIGN KEY (`language_id`) REFERENCES `sakila`.`language` (`language_id`) ON DELETE RESTRICT ON UPDATE CASCADE                   |
| ALTER TABLE `sakila`.`film` ADD CONSTRAINT `fk_film_language_original` FOREIGN KEY (`original_language_id`) REFERENCES `sakila`.`language` (`language_id`) ON DELETE RESTRICT ON UPDATE CASCADE |
| ALTER TABLE `sakila`.`film_actor` ADD CONSTRAINT `fk_film_actor_actor` FOREIGN KEY (`actor_id`) REFERENCES `sakila`.`actor` (`actor_id`) ON DELETE RESTRICT ON UPDATE CASCADE                   |
| ALTER TABLE `sakila`.`film_actor` ADD CONSTRAINT `fk_film_actor_film` FOREIGN KEY (`film_id`) REFERENCES `sakila`.`film` (`film_id`) ON DELETE RESTRICT ON UPDATE CASCADE                       |
| ALTER TABLE `sakila`.`film_category` ADD CONSTRAINT `fk_film_category_category` FOREIGN KEY (`category_id`) REFERENCES `sakila`.`category` (`category_id`) ON DELETE RESTRICT ON UPDATE CASCADE |
| ALTER TABLE `sakila`.`film_category` ADD CONSTRAINT `fk_film_category_film` FOREIGN KEY (`film_id`) REFERENCES `sakila`.`film` (`film_id`) ON DELETE RESTRICT ON UPDATE CASCADE                 |
| ALTER TABLE `sakila`.`inventory` ADD CONSTRAINT `fk_inventory_film` FOREIGN KEY (`film_id`) REFERENCES `sakila`.`film` (`film_id`) ON DELETE RESTRICT ON UPDATE CASCADE                         |
| ALTER TABLE `sakila`.`inventory` ADD CONSTRAINT `fk_inventory_store` FOREIGN KEY (`store_id`) REFERENCES `sakila`.`store` (`store_id`) ON DELETE RESTRICT ON UPDATE CASCADE                     |
| ALTER TABLE `sakila`.`payment` ADD CONSTRAINT `fk_payment_customer` FOREIGN KEY (`customer_id`) REFERENCES `sakila`.`customer` (`customer_id`) ON DELETE RESTRICT ON UPDATE CASCADE             |
| ALTER TABLE `sakila`.`payment` ADD CONSTRAINT `fk_payment_rental` FOREIGN KEY (`rental_id`) REFERENCES `sakila`.`rental` (`rental_id`) ON DELETE SET NULL ON UPDATE CASCADE                     |
| ALTER TABLE `sakila`.`payment` ADD CONSTRAINT `fk_payment_staff` FOREIGN KEY (`staff_id`) REFERENCES `sakila`.`staff` (`staff_id`) ON DELETE RESTRICT ON UPDATE CASCADE                         |
| ALTER TABLE `sakila`.`rental` ADD CONSTRAINT `fk_rental_customer` FOREIGN KEY (`customer_id`) REFERENCES `sakila`.`customer` (`customer_id`) ON DELETE RESTRICT ON UPDATE CASCADE               |
| ALTER TABLE `sakila`.`rental` ADD CONSTRAINT `fk_rental_inventory` FOREIGN KEY (`inventory_id`) REFERENCES `sakila`.`inventory` (`inventory_id`) ON DELETE RESTRICT ON UPDATE CASCADE           |
| ALTER TABLE `sakila`.`rental` ADD CONSTRAINT `fk_rental_staff` FOREIGN KEY (`staff_id`) REFERENCES `sakila`.`staff` (`staff_id`) ON DELETE RESTRICT ON UPDATE CASCADE                           |
| ALTER TABLE `sakila`.`staff` ADD CONSTRAINT `fk_staff_address` FOREIGN KEY (`address_id`) REFERENCES `sakila`.`address` (`address_id`) ON DELETE RESTRICT ON UPDATE CASCADE                     |
| ALTER TABLE `sakila`.`staff` ADD CONSTRAINT `fk_staff_store` FOREIGN KEY (`store_id`) REFERENCES `sakila`.`store` (`store_id`) ON DELETE RESTRICT ON UPDATE CASCADE                             |
| ALTER TABLE `sakila`.`store` ADD CONSTRAINT `fk_store_address` FOREIGN KEY (`address_id`) REFERENCES `sakila`.`address` (`address_id`) ON DELETE RESTRICT ON UPDATE CASCADE                     |
| ALTER TABLE `sakila`.`store` ADD CONSTRAINT `fk_store_staff` FOREIGN KEY (`manager_staff_id`) REFERENCES `sakila`.`staff` (`staff_id`) ON DELETE RESTRICT ON UPDATE CASCADE                     |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Most of the foreign key constraints use RESTRICT for DELETE (meaning you are not allowed to delete a parent row when children exist), and CASCADE for UPDATE (meaning changes to parent will propagate to children). This is good, since I want to test behavior of both RESTRICT and CASCADE.

OK, we wish to remove these constraints from the slave. To see what we are going to do, consider:

slave1> select drop_statement from common_schema.sql_foreign_keys where table_schema='sakila';
+-----------------------------------------------------------------------------------+
| drop_statement                                                                    |
+-----------------------------------------------------------------------------------+
| ALTER TABLE `sakila`.`address` DROP FOREIGN KEY `fk_address_city`                 |
| ALTER TABLE `sakila`.`city` DROP FOREIGN KEY `fk_city_country`                    |
| ALTER TABLE `sakila`.`customer` DROP FOREIGN KEY `fk_customer_address`            |
| ALTER TABLE `sakila`.`customer` DROP FOREIGN KEY `fk_customer_store`              |
| ALTER TABLE `sakila`.`film` DROP FOREIGN KEY `fk_film_language`                   |
| ALTER TABLE `sakila`.`film` DROP FOREIGN KEY `fk_film_language_original`          |
| ALTER TABLE `sakila`.`film_actor` DROP FOREIGN KEY `fk_film_actor_actor`          |
| ALTER TABLE `sakila`.`film_actor` DROP FOREIGN KEY `fk_film_actor_film`           |
| ALTER TABLE `sakila`.`film_category` DROP FOREIGN KEY `fk_film_category_category` |
| ALTER TABLE `sakila`.`film_category` DROP FOREIGN KEY `fk_film_category_film`     |
| ALTER TABLE `sakila`.`inventory` DROP FOREIGN KEY `fk_inventory_film`             |
| ALTER TABLE `sakila`.`inventory` DROP FOREIGN KEY `fk_inventory_store`            |
| ALTER TABLE `sakila`.`payment` DROP FOREIGN KEY `fk_payment_customer`             |
| ALTER TABLE `sakila`.`payment` DROP FOREIGN KEY `fk_payment_rental`               |
| ALTER TABLE `sakila`.`payment` DROP FOREIGN KEY `fk_payment_staff`                |
| ALTER TABLE `sakila`.`rental` DROP FOREIGN KEY `fk_rental_customer`               |
| ALTER TABLE `sakila`.`rental` DROP FOREIGN KEY `fk_rental_inventory`              |
| ALTER TABLE `sakila`.`rental` DROP FOREIGN KEY `fk_rental_staff`                  |
| ALTER TABLE `sakila`.`staff` DROP FOREIGN KEY `fk_staff_address`                  |
| ALTER TABLE `sakila`.`staff` DROP FOREIGN KEY `fk_staff_store`                    |
| ALTER TABLE `sakila`.`store` DROP FOREIGN KEY `fk_store_address`                  |
| ALTER TABLE `sakila`.`store` DROP FOREIGN KEY `fk_store_staff`                    |
+-----------------------------------------------------------------------------------+

To actually make the DROP, we use common_schema's eval():

slave1> call common_schema.eval("select drop_statement from common_schema.sql_foreign_keys where table_schema='sakila'");

eval() is a handy routine which invokes statements generated by the given query.

This concludes the setup part.

Tests will include:

1. Attempting to delete a parent row
2. Attempting to add an invalid child row
3. Attempting to update parent row

I was thinking there would be a difference between the two binary log file formats: STATEMENT and ROW. But the tests I produced showed no difference.

Tests

Attempting to delete parent row:

master> delete from actor where actor_id=1;
ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`sakila`.`film_actor`, CONSTRAINT `fk_film_actor_actor` FOREIGN KEY (`actor_id`) REFERENCES `actor` (`actor_id`) ON UPDATE CASCADE)

slave1> select * from actor where actor_id=1;
+----------+------------+-----------+---------------------+
| actor_id | first_name | last_name | last_update         |
+----------+------------+-----------+---------------------+
|        1 | PENELOPE   | GUINESS   | 2006-02-15 04:34:33 |
+----------+------------+-----------+---------------------+

Good: the master refused the DELETE, and no DELETE occurred on slave. Integrity is intact.

Attempting to add an invalid child row:

master> insert into film_actor (actor_id, film_id, last_update) values (9999, 1, NOW());
ERROR 1452 (23000): Cannot add or update a child row: a foreign key constraint fails (`sakila`.`film_actor`, CONSTRAINT `fk_film_actor_actor` FOREIGN KEY (`actor_id`) REFERENCES `actor` (`actor_id`) ON UPDATE CASCADE)

slave> select * from film_actor where actor_id=9999;
Empty set (0.00 sec)

Integrity is still intact.

Attempting to update parent row: there is nothing invalid about this operation. I'm wondering whether changes are CASCADEd on slave as well as on master:

master> update actor set actor_id=999 where actor_id=199;

master> select count(*) from film_actor where actor_id=999;
+----------+
| count(*) |
+----------+
|       15 |
+----------+

The 999 value wasn't there before on the master, so this verifies the CASCADE works on master. As for slave:

slave> select count(*) from actor where actor_id=999;
+----------+
| count(*) |
+----------+
|        1 |
+----------+

slave> select count(*) from film_actor where actor_id=999;
+----------+
| count(*) |
+----------+
|        0 |
+----------+

Bummer! The actor's row was updated, but cascading did not work on slave.

This is actually documented. However, the documentation only relates to the issue of slave tables being MyISAM. The problem occurs even when the slave tables are InnoDB, and have no foreign key constraints.

Conclusion

My personal interest in the scenario is due to something I'm working on, I'll elaborate on a future post. People sometime hope to get rid of foreign keys, and might wonder whether replication performance would boost having constraints removed on slaves.

When slave does not enforce foreign keys, you cannot rely on integrity with cascading constraints. An ugly patch might be to use triggers so as to simulate their behavior. Performance wise this is very bad.

‫אתמול פרסמתי פוסט על רצוני להימנע מרכישת שרתים מהיצרנים הגדולים ועל הרצון לבנות שרתים בעצמי. בעקבות כמה טוקבקים שנכתבו שם, חשבתי להבהיר כמה דברים הן על "חץ ביז" והן על כיוון העסק ולגבי יצור שרתים: הפוקוס של "חץ ביז" תמיד היה ותמיד יהיה שרתי VPS (שרתים וירטואליים) בארץ או בחו"ל. זה היה וזה יהיה התחום [...]‬

Other highlights

Of course, the Turkic Wikimedia Conference had many other highlights except my talks and workshops. Jonas Öberg from Creative Commons delivered a keynote speech about the importance of letting people freely share their works, especially with regards to cultures which are not as known as the American or the Western European, such as that of Kazakhstan. Basically, anybody who is curious about the culture of Kazakhstan will only be able to know about it the things that are freely posted online. If it’s gathering dust in the library or locked behind a password in a pay-to-read website, nobody will read it.

The Wikimedian Daniel Mietchen, who is an advocate for Open Science, convincingly explained why opening up academic articles and experiments will not just make them cheaper, but also more correct scientifically.

Daniel also impressed lots of people with his Russian speaking skills: Apparently, he grew up in East Germany, where all children had to study Russian in schools, and he was one of the few children who actually bothered to learn it well. He said that at first he didn’t like to be forced to learn a language that wasn’t useful to him, but when he had to read a book of prose – The Tales of the Late Ivan Petrovich Belkin – as homework, he found it very satisfying, even though it was very hard in the beginning.

Another highlight was a book about editing Wikipedia given to me by one of its authors Irada Alakbarova, a participant from Azerbaijan. It is similar in content and scope to the book written by the French Wikimedians Guillaume Paumier and Florence Devouard, but it’s impressive that Irada is not just an enthusiastic Wikimedian, but also a department head in the Information Technology Institute of the Azerbaijan Academy of Sciences, and the book’s other author Rasim Aliquliyev is the Institute’s director. (In precise Azeri spelling their names are İradə Ələkbərova and Rasim Əliquliyev. The letter Ə is a part of Azerbaijan’s Latin-based writing system, but looks too weird to many English readers.)

Irada also told me that some time ago she gathered any information that she could about Wikipedia’s server configuration and used it as an example for teaching configuration of high-performance websites. She was very happy when I told that the Wikimedia server configuration became even more transparent recently.

Summing up

I participated in many conferences lately, and this one was unusually satisfying in many ways.

As usual, meeting the people was the best part. This refers both to the people from places like Bashkortostan and Sakha, with whom I communicated by email for many years, hardly imagining how do they look, and also to people whom I had not known before and who came from countries that I could hardly imagine of ever visiting, like Kyrgyzstan and Turkmenistan. The international press mostly reports bad and weird news from these countries, but as it often happens, the image created by the media has little to do with the real people – I was stunned by the talent, the originality and the vigor that they demonstrated.

I was not the only one who felt that the conference was a great success, so we already started to throw around ideas for the location of another one. The names of Bishkek, Ufa, Baku and Istanbul were suggested, and I would certainly be very happy to go to any of these cities or to meet these wonderful people elsewhere.

Most importantly, this conference left me and the other participants a long list of exciting tasks to do.

Filed under: Free Software, translation, Wikipedia Tagged: Creative Commons, Open Science

‫השבוע היה בתכנון להזמין עבור העסק מספר שרתים נוספים מ-HP להרחבת העסק. בדקנו מחירים, התווכחנו עם משווקים, קיבלנו הצעות מספקים שונים והמחירים כרגיל הראו שהמשווקים פה מרוויחים בכלל לא רע על חשבון עסקים כמו חץ ביז. אז לפני שאני שם חותמת עסק וחותם על ההזמנה החלטתי קצת לחשוב מחדש על עניין ההזמנה. בכל זאת, מדובר [...]‬

OSM, הגוף שעומד מאוחרי Joomla! הודיע כי לועד המנהל שלו נבחרו שלושה חברים חדשים. אחד מהם, עופר כהן שמו, פעיל בקהילת ה-Joomla! הישראלית ואחד מהמארגנים של יום ג'ומלה 2011.

בנוסף לתרומת הקוד לפרוייקט ולתרגום לעברית, עופר גם הולך להעביר הרצאה בנושא "Joomla architecture revealed" בכנס הג'ומלה הבינלאומ "J and Beyond". כל הכבוד עופר ובהצלחה (:

Filed under: Free Software in Israel, Israeli Community

חומרים ב-Cycles כל הרינדורים עם תאורת HDRI ו100 דגימות. Background = רקע, לרוב זה אומר שאין שום חומר אלא רק שקיפות. Diffuse = חומר אטום ללא השתקפות, מאפיין אובייקטים כמו ריצפה, חול, קירות וכו'.    Glossy = חומר אטום ומבריק, מאפיין חומרים כמו כסף, זהב, מראה וכו'. Glass = מאפיין אובייקטים בעלי שקיפות, מתאים לחומרים כמו מים, זכוכית, חלונות וכו'. Translucent =

‫הכל התחיל לפני מספר ימים. דבורית שרגל (בעלת הבלוג Velvet Underground שמתארח אגב, אצלנו) כתבה בקצרה על בעיה שיש לה: כאחת שנמצאת בתחום התקשורת, היא מקבלת לעיתים בקשות "דרושים" – חברות ועסקים בתחום התקשורת מחפשים אנשים מסויימים והיא מפרסמת את המודעות אצלה באתר (כאן). דבר מעולה, לדעתי. רק ישנה בעיה קטנה: כמו שקורה במקרים רבים, תמיד [...]‬

There’s something that has worried me greatly ever since my ISP came to install my router upgrade.

I won’t mention names, but it seems certain providers seem to favour setting the WPA-PSK passphrase to the phone number of the customer by default.
I have noticed that the ESSID is sometimes a form of the customer surname too.

This is of course dismal security-wise.
Firstly, if I can somehow get the target’s phone number (e.g. looking it up in the phonebook DUH), I can probably just guess the passphrase.

Secondly, even if I have no idea what the phone number is, by using this ‘convention’ for passphrases, the providers have GREATLY reduced the keyspace – making brute force attacks not only possible but practical too.

Let’s take a look at HOT for example. Their phone number-space is 077-XXX-XXXX.
That’s only 10,000,000 possible keys!

The following benchmarks from the Pyrit project paint a pretty bleak picture:

Now I haven’t benchmarked my home system yet, but lets conservatively say that my Radeon HD 5670 can pull a modest 5k PMKs/s I could cover the entire keyspace in 2000s. Statistically speaking, I’d likely hit the key in under 17 minutes!

Pretty practical I’d say.

So basically, providers; please stop doing stupid things like this!

Another technical issue that bothered many people in the Turkic Wikimedia Conference in Almaty is support for their language in Google Translate. Though this is not directly related to Wikimedia, I was asked about this repeatedly by the participants, as well as by local journalists who interviewed me. Some people even referred to it as a “conspiracy”.

X

Luckily, one of the participants was Tilek Mamutov, a Google employee from Kyrgyzstan, and he delivered a whole talk about it. His main message was that there is no conspiracy, and that to support more languages Google mostly needs to process as many texts as possible in that language, if possible – with a parallel translation. There are much less digital texts in languages like Kyrgyz and Bashkir than there are in German and Spanish, so it is not yet possible.

However, there is hope: a group of volunteers in Kyrgyzstan is working on creating a database of digital translated texts with the specific goal of making it usable in Google Translate. WikiBilim, the Kazakh association that organized the conference works on a similar initiative, too.

On my behalf, I suggested a convenient way to gather texts in these languages: to upload literature in them to Wikisource. I also mentioned the existence of Apertium. Apertium is a Free machine translation engine, which can be adapted to any language. It was developed in Valencia, and the first languages that it started to support are languages that are relevant for Spain: Spanish, Catalan, Basque, English and also the closely-related Esperanto, and it translates between them quite well. It supports a few other languages, too.

And it can support even more languages. Like Google Translate, it also needs as many digital texts as possible to actually start working, and it also It needs dictionaries and tables of grammar rules, because it tries several methodologies for translation. Work has already begun for Turkish-Azeri and Turkish-Kyrgyz, and there are projects for Turkish-Chuvash and other language pairs. All these projects need people who can test them, contribute words to the dictionaries and check the grammar rules. So if you want to help complete a Free Turkish-Azeri machine translation system or to create an English-Kyrgyz translation system, contact the Apertium project.

To be continued…

---

Oh (edit): A correction came from Apertium developers: Apertium *doesn’t* need any texts, except for testing purposes. The more texts we have, the more we can test, of course, but above all, we need native speakers of languages who understand the grammar of the languages they’re working on and can work with computational formalisms.

Filed under: Free Software, Google, language, translation, Wikipedia Tagged: Apertium, Chuvash, digital texts, Kyrgyz, parallel translation, Turkish, wikisource

I installed Ubuntu 12.04 on two laptops by now. and I must rant. Ubuntu developers are assuming two many things about me. First, there is this bug, which the 'migration-assitant' mounts existing partitions in READ-WRITE mode. I find it dangerous and not necessary for migration.
Then, today I installed Ubuntu 12.04 on IBM x121e and found out that my Bluetooth chip is disabled. After about 20 min of looking I discovered, it is because of a bug in the installer which messes around with the BIOS. So if you have IBM x120e with Intel Chipset, you need to reboot, restore the BIOS defaults and then Bluetooth works. 
That's all the ranting for now ...

I have a weired hobby, checking very frequently the Release-Critical Bugs in Debian.
It's kind of a magic window to all this wonderful packages found in Debian. Sometimes I'm horrified to see so many bugs opened at once. Like now ... 169 FTBS introduced by GCC-4.7...



It's no reason to panic, but it's been a while since I've seen such a huge bump up...

Another technical issue that bothered many people in the Turkic Wikimedia Conference in Almaty is support for their language in Google Translate. Though this is not directly related to Wikimedia, I was asked about this repeatedly by the participants, as well as by local journalists who interviewed me. Some people even referred to it as a “conspiracy”.

Luckily, one of the participants was Tilek Mamutov, a Google employee from Kyrgyzstan, and he delivered a whole talk about it. His main message was that there is no conspiracy, and that to support more languages Google mostly needs to process as many texts as possible in that language, if possible – with a parallel translation. There are much less digital texts in languages like Kyrgyz and Bashkir than there are in German and Spanish, so it is not yet possible.

However, there is hope: a group of volunteers in Kyrgyzstan is working on creating a database of digital translated texts with the specific goal of making it usable in Google Translate. WikiBilim, the Kazakh association that organized the conference works on a similar initiative, too.

On my behalf, I suggested a convenient way to gather texts in these languages: to upload literature in them to Wikisource. I also mentioned the existence of Apertium. Apertium is a Free machine translation engine, which can be adapted to any language. It was developed in Valencia, and the first languages that it started to support are languages that are relevant for Spain: Spanish, Catalan, Basque, English and also the closely-related Esperanto, and it translates between them quite well. It supports a few other languages, too.

And it can support even more languages. Like Google Translate, it also needs as many digital texts as possible to actually start working, and it also needs dictionaries and tables of grammar rules, because it tries several methodologies for translation. Work already begun for Turkish-Azeri and Turkish-Kyrgyz, and there are projects for Turkish-Chuvash and other language pairs. All these projects need people who can test them, contribute words to the dictionaries and check the grammar rules. So if you want to help complete a Free Turkish-Azeri machine translation system or to create an English-Kyrgyz translation system, contact the Apertium project.

To be continued…

Filed under: Free Software, Google, language, translation, Wikipedia Tagged: Apertium, Chuvash, digital texts, Kyrgyz, parallel translation, Turkish, wikisource

Two git tips: list remote repo; show direct merge path

Listing remote git repositories

Sometimes public git repositories have no convenient web interface. Their existence can be inferred from a mention in a git merge commit, like this one. To get a list of branches in a repository using the native git protocol, use the git ls-remote command:

git ls-remote -h git://sources.calxeda.com/kernel/

Use -t to get tags listing.

Show merge path log

By default git log does not necessarily reflect the actual "ancestry" line of commits. To know which merges a given commit went through on its way to the main development tree, do

git log --ancestry-path commit1..commit2

where commit1 is the commit you are interested in, and commit2 is any random later commit, such as a version tag.

Two git tips: list remote repo; show direct merge path

Listing remote git repositories

Sometimes public git repositories have no convenient web interface. Their existence can be inferred from a mention in a git merge commit, like this one. To get a list of branches in a repository using the native git protocol, use the git ls-remote command:

git ls-remote -h git://sources.calxeda.com/kernel/

Use -t to get tags listing.

Show merge path log

By default git log does not necessarily reflect the actual "ancestry" line of commits. To know which merges a given commit went through on its way to the main development tree, do

git log --ancestry-path commit1..commit2

where commit1 is the commit you are interested in, and commit2 is any random later commit, such as a version tag.

jQury board היא ספרית javascript מבוססת jQuery-ui. הספריה מאפשרת ליצור לוח עבודה עליו אפשר להניח עצמים שונים בעלי התנהגויות שונות. במצב עריכה, ניתן לגרור את העצמים על הלוח ולערוך אותם בצורה אינטראקטיבית. במצב צפיה, העצמים אינם אינטראקטיביים, וניתן לצפות בהם ולשנות פרמטרים שונים בעזרת פקודות הספריה.

Ubuntu "engineers" are probably dumber than that Enterprise Linux OS. I got a laptop from a friend who asked me to install the latest Ubuntu on it. My usual technique of make a live bootable usb faild.
So, I googled and found this evidence that Ubuntu engineers are retarded.
You need Ubuntu to install Ubuntu.
This reminds me a bit that Redmond company. What if I have another OS ? NOT Mac or Windows ? For example, Debian? can't you just write how to obtain that package ???

קפצתי היום למפגש בלנדר לראות מי יגיע. חשבתי שצפוי מפגש קטן של 5-10 משוגעים לעניין וכשהגעתי גיליתי שיש בחדר 50 איש, ואז אחרי שכמה עזבו. גם מגוון האנשים מרשים ביותר. והחלק הכי מעניין/מפתיע הוא שאת רובם אני לחלוטין לא מכיר מקהילת הקוד הפתוח בארץ והמפגשים/כנסים שלה.

מאחר ואין לי קשר לבלנדר או לנושא גרפיקה בתלת מימד, רוב ההרצאות נשמעו לי כמו סינית, אבל מהתרשמותי לשאר האנשים, שכן בעניינים, היה מאוד מעניין. דיברתי קצת עם המארגנים כדי לראות איך מנצלים את התנופה ומנסים ללכת לכיוון של גיבוש קהילה ומסורת של מפגשים והחלפת ידע. אומנם אנחנו רק בהתחלה, אך צריך לחשוב קדימה.

Filed under: Free Software in Israel, Israeli Community

קפצתי היום למפגש בלנדר לראות מי יגיע. חשבתי שצפוי מפגש קטן של 5-10 משוגעים לעניין וכשהגעתי גיליתי שיש בחדר 50 איש, ואז אחרי שכמה עזבו. גם מגוון האנשים מרשים ביותר. והחלק הכי מעניין/מפתיע הוא שאת רובם אני לחלוטין לא מכיר מקהילת הקוד הפתוח בארץ והמפגשים/כנסים שלה.

מאחר ואין לי קשר לבלנדר או לנושא גרפיקה בתלת מימד, רוב ההרצאות נשמעו לי כמו סינית, אבל מהתרשמותי לשאר האנשים, שכן בעניינים, היה מאוד מעניין. דיברתי קצת עם המארגנים כדי לראות איך מנצלים את התנופה ומנסים ללכת לכיוון של גיבוש קהילה ומסורת של מפגשים והחלפת ידע. אומנם אנחנו רק בהתחלה, אך צריך לחשוב קדימה.

Filed under: Free Software in Israel, Israeli Community

Yesterday I attended the Zero-Downtime Schema Changes In MySQL webinar by Baron Schwartz, Percona (do you say "attended" for something you listened to from your home office?)

I was keen to learn about possible enhancements and improvements of pt-online-schema-change over oak-online-alter-table. Here are my impressions:

The base logic of pt-online-schema-change is essentially the same as of oak-online-alter-table. You create a ghost/shadow table, create complex triggers, copy in chunks, freeze and swap. Both work on any type of PRIMARY KEY (oak-online-alter-table can work with any UNIQUE KEY, I'm not sure about pt-online-schema-change on this), be it an INTEGER, other type, or a multi column one.

However, pt-online-schema-change also adds the following:

• It supports FOREIGN KEYs (to some extent). This is something I've wanted to do with oak-online-alter-table but never got around to it. Foreign keys are very tricky, as Baron noted. With child-side keys, things are reasonably manageable. With parent-side this becomes a nightmare, sometimes unsolvable (when I say "unsolvable", I mean that under the constraint of having the operation run in a non-blocking, transparent way).
• Chunk size is auto-calculated by the script. This is a cool addition. Instead of letting the user throwing out numbers like 1,000 rows per chunk, in the hope that this is neither too small nor too large, the tool monitors the time it takes a chunk to complete, then adjusts the size of next chunk accordingly. Hopefully this leads to a more optimized run, where locks are only held for very short periods, yet enough rows are being processed at a time.
• The tool looks into replicating slaves to verify they're up to the job. If the slave lags too far, the tool slows down the work. This is an excellent feature, and again, one that I always wanted to have. Great work!

So the three bullets above are what I understand to be the major advantages of Percona's tool over oak-online-alter-table.

Q & A

The presentation itself was very good, and Baron answered some questions. There was one question he did not answer during the webinar, nor here, and I though I may pop in and answer it. Although I can't speak for the coders of pt-online-schema-change, I safely assume that since the logic follows that of oak-online-alter-table, the same answer applies in the case of Percona's toolkit.

But, first, a background question (asked and answered during the webinar):

Q: What if my table already has AFTER TRIGGERs?

A: Then this can't work out. The table must not have triggers.

Which led to the next question:

Q: Can't the tool use BEFORE TRIGGERs instead?

Imagine a MyISAM table being altered to InnoDB (this is a major task for which my tool was built). Suppose we used a BEFORE trigger on an INSERT, but the INSERT failed. That would make the shadow table inconsistent with the original table. Which is the reason why the trigger must be an AFTER trigger.

With InnoDB this should not be an issue, since triggers and actions all play within the same transaction, so all succeed or all fail. I have this nagging feeling at the back of my head which says I've already had thoughts on this and have found a problem with InnoDB tables as well. I can't put my finger on it now, so no comment on this one at this stage.

After a long period of yak shaving, yesterday I decided to go for the short-cut and set up a minimal system for the Perl Maven web site.

For the full article visit Perl Maven Cookbook

For years OmniCppComplete has been the de facto standard for C++ completion in Vim. But as time progressed, I got more and more annoyed by it’s shortcomings. OmniCppComplete is based on tokenizing provided by ctags. The ctags parsing of C++ code is problematic, you can’t even run it on libstdc++ headers (you need to download modified headers). You want to use an external library? You’ll need to run ctags seperatly on each library. Not to mention it’s inablity to deduce types of anything more than trivial. The core of the problem is that OmniCppComplete isn’t a compiler and you can’t expect something that isn’t a compiler to fully understand code. This what makes Visual Studio’s IntelliSense so great: it uses the Visual C++ compiler for parsing, it isn’t making wild guess at types and what is the current scope – it knows it.

With this on my mind, I set out to look for a better alternative. The first thing I looked for is getting GCC to give me type info. Apparently, GCC doesn’t like to play nice with other and provide this information (I read in couple of places people cite RMS/FSF saying it’s deliberately in order not to allow proprietary software to use GCC without linking against it, like crashing your car so nobody wants to steal it – plain dumb). The only solution is to patch like GCCSense, which does exactly what I looked for, compiler based auto-completion. But patching GCC only to work around perverted FSF policies seems like not the most elegant solution. This is where clang, a modular and developer friendly compiler, came to my mind.

After a quick searching I came across clang complete. This plugin uses clang to parse your code and suggest completion. Auto-complete worked out of the box for the standard library, STL and Boost, and it worked properly (already better than OmniCppComplete). If you want to use libraries which aren’t in your default search path, i.e. gcc need you to specify -I flags or you need -D flags, you’ll need to pass them on to clang. This may sound difficult, but the developers of the vim script provide a wrapper to automatically understand what should be the proper flags from your build system. All you need to do is pass CXX='~/.vim/bin/cc_args.py g++' to make and the script will record your flags in a .clang_complete file, which it will use from now on. After you do it (except a little glitch, see bellow), omni-completion will work for every library in your code base.

There are still some glitches that need to be polished. First of all, the suggested way to invoke cc_args.py isn’t perfect. Make changes working dirs as it recurses through your code base, so cc_args.py‘s working-dir isn’t the same as yours, but the plugin will search for the .clang_complete file in your working-dir (the one vim has). The solution is just to move the .clang_complete files to the main directory of the project, from whatever subdirectory it created them in, and merge them (simpler than it sounds). Also this task needs to be done once for every project, so no big deal.

There are two other problems that still bother me. The plugin’s author says that if you set the plugin to use libclang it would work much faster and use caching. However, libclang isn’t currently packaged for Ubuntu 11.10. This should be solve next month with the release of Ubuntu 12.04, which provides it. The second problem is a weird bug in clang. When I try to compile a C++11 file with clang (providing the -std=c++0x flag, it segfaults, pretty weird. But I guess a bit more searching and maybe some bug-reporting will fix this issue (sadly, this means that the plugin can’t autocomplete c++11 features as well).

Despite some rough edges it’s way better than any previous C++ completion for Vim I ever seen. At the begging I hesitated when I had to remove OmniCppComplete due to conflicts with clang complete, and now after a (very) shot time using clang complete I don’t think I’ll miss it anytime soon.

Update May 2, 2012: Ubuntu 12.04 got released, so I finally checked the status of the completion using libclang. Apparently, you also need to install the libclang-dev package (in addition to libclang1, or else, clang_complete will output cryptic error messages. The clang 3.0 that comes with Ubuntu 12.04 no longer segfaults on c++11 code, but it refuses the compile anything with boost’s shared_ptr, due to issues with move semantics.