Tuesday, September 30, 2014

Packt Publishing - ALL eBooks and Videos are just $10 each or less until the 2nd of October

Just spread good campaign from Packt Publishing - It's a good news for people who love to learn something new - ALL eBooks and Videos are just $10 or less -- the more you choose to learn, the more you save:
  • Any 1 or 2 eBooks/Videos -- $10 each
  • Any 3-5 eBooks/Videos -- $8 each
  • Any 6 or more eBooks/Videos -- $6 each


Saturday, September 27, 2014

I Heart Logs - Event Data, Stream Processing, and Data Integration by Jay Kreps; O'Reilly Media

As I have worked in server-side a long time as System Administrator. I must spend with logs. To use it for checking and investigation in issue. As some policies in some Companies, they want to keep logs over year or over ten years. So, it is not unusual to find out idea to store, integrate logs and do something.
A book tittle "I Heart Logs - Event Data, Stream Processing, and Data Integration" by Jay Kreps. It's very interesting. I'd like to know what I can learn from it, how logs work in distributed systems and learn from author who works at LinkedIn. A book! Not much for the number of pages. However, it gives much more for data flow idea, how logs work and author still shows readers why logs are worthy of reader's attention. In a book, that has only 4 chapters, but readers will get concept and idea about Data integration (Making all of an organization’s data easily available in all its storage and processing systems), Real-time data processing (Computing derived data streams) and Distributed system design (How practical systems can by simplified with a log-centric design). In addition, I like it. because author wrote from his experience at LinkedIn.

After reviewing: A book refers a lot of information(It's easy on ebook to click links) that's useful. Readers can use them and find out more on the Internet and use. For Data integration, It's focused to Kafka software that is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system. Additional, It gave why the Big Data Lambda Architecture is good for batch system and a stream processing system and point about things a log can do.

So, Readers will be able to learn:
  • Learn how logs are used for programmatic access in databases and distributed systems
  • Discover solutions to the huge data integration problem when more data of more varieties meet more systems
  • Understand why logs are at the heart of real-time stream processing
  • Learn the role of a log in the internals of online data systems
  • Explore how Jay Kreps applies these ideas to his own work on data infrastructure systems at LinkedIn
Book - I Heart Logs - Event Data, Stream Processing, and Data Integration
Author: Jay Kreps

Monday, September 22, 2014

Where is my space on Linux filesystem?

Not Often, I checked about my space after made filesystem on Linux. Today, I have made Ext4 filesystem around 460GB, I found it 437GB only. Some path should be 50GB, but it was available only 47GB.
Thank You @OracleAlchemist and @gokhanatil for good information about it.
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup0-U01LV   50G   52M   47G   1% /u01

Reference  - It's for specify the percentage of the filesystem blocks reserved for the super-user. This avoids fragmentation, and allows root-owned daemons, such as syslogd(8), to continue to function correctly after non-privileged processes are prevented from writing to the  filesystem. The default percentage is 5%.
After I found out more information. Look like we can set it to zero, but we should not set it to zero for /,/var,/tmp or which path has lots of file creates and deletes.
If you set the reserved block count to zero, it won't affect
performance much except if you run for long periods of time (with lots
of file creates and deletes) while the filesystem is almost full
(i.e., say above 95%), at which point you'll be subject to
fragmentation problems.  Ext4's multi-block allocator is much more
fragmentation resistant, because it tries much harder to find
contiguous blocks, so even if you don't enable the other ext4
features, you'll see better results simply mounting an ext3 filesystem
using ext4 before the filesystem gets completely full.
If you are just using the filesystem for long-term archive, where
files aren't changing very often (i.e., a huge mp3 or video store), it
obviously won't matter.
- Ted
Example: Changed reserved-blocks-percentage 
[root@mytest01 u01]# df -h /u01
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup0-U01LV   50G   52M   47G   1% /u01
[root@mytest01 u01]# tune2fs -m 1 /dev/mapper/VolGroup0-U01LV
tune2fs 1.43-WIP (20-Jun-2013)
Setting reserved blocks percentage to 1% (131072 blocks)
[root@mytest01 u01]# df -h /u01
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup0-U01LV   50G   52M   49G   1% /u01
[root@mytest01 u01]# tune2fs -m 5 /dev/mapper/VolGroup0-U01LV
tune2fs 1.43-WIP (20-Jun-2013)
Setting reserved blocks percentage to 5% (655360 blocks)
[root@mytest01 u01]# df -h /u01
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup0-U01LV   50G   52M   47G   1% /u01
Finally, I knew it was reserved for super-user. Checked more for calculation.
[root@ottuatdb01 ~]# df -m /u01
Filesystem                  1M-blocks  Used Available Use% Mounted on
/dev/mapper/VolGroup0-U01LV     50269    52     47657   1% /u01
[root@ottuatdb01 ~]#  tune2fs -l /dev/mapper/VolGroup0-U01LV |egrep  'Block size|Reserved block count'
Reserved block count:     655360
Block size:               4096

Available = 47657MB
Used = 52M
Reserved Space = (655360 x 4096) / 1024 /1024 = 2560MB 
Total = 47657 + 2560 + 52 = 50269 

OK.. I felt good after it cleared for me. Somehow, I believe On Hug space, 5% of the filesystem blocks reserved that's too much. We can reduce it.

Other Links:
https://www.redhat.com/archives/ext3-users/2009-January/msg00026.html
http://unix.stackexchange.com/questions/7950/reserved-space-for-root-on-a-filesystem-why
http://linux.die.net/man/8/tune2fs
https://wiki.archlinux.org/index.php/ext4#Remove_reserved_blocks