четверг, декабря 20, 2012

Starting mongod: error command line: multiple occurrences

I tried to start mongod service and got error:

[root@mongo root]# service mongod start
Starting mongod: error command line: multiple occurrences
use --help for help
                                                           [  OK  ]

service didn't start an didn't log nothing
problem was into /etc/mongod.conf
I set one of parameters double into config, in my case it was "fork = true"
after removed one, service started successfully

среда, ноября 07, 2012

Symfony 2.1 SonataUserBundle errors

error at http://localhost/admin/sonata/user/user/create:  
The server returned a "500 Internal Server Error".

 log:
request.INFO: Matched route "admin_sonata_user_user_create" (parameters: "_controller": "Sonata\AdminBundle\Controller\CRUDController::createAction", "_sonata_admin": "sonata.user.admin.user", "_sonata_name": "admin_sonata_user_user_create", "_route": "admin_sonata_user_user_create")
...
 request.CRITICAL: RuntimeException: The locale resource bundle could not be loaded for locale "en" (uncaught exception) at vendor/symfony/symfony/src/Symfony/Component/Locale/Locale.php line 160

solution:
>> php vendor/symfony/symfony/src/Symfony/Component/Locale/Resources/data/build-data.php 49


error at http://localhost/admin/sonata/user/user/list:
The server returned a "500 Internal Server Error".

log:
request.INFO: Matched route "admin_sonata_user_user_list" (parameters: "_controller": "Sonata\AdminBundle\Controller\CRUDController::listAction", "_sonata_admin": "sonata.user.admin.user", "_sonata_name": "admin_sonata_user_user_list", "_route": "admin_sonata_user_user_list")
...
request.CRITICAL: Twig_Error_Runtime: An exception has been thrown during the rendering of a template ("None of the chained routers were able to generate route "homepage".") in "SonataAdminBundle:CRUD:list.html.twig". (uncaught exception) at app/cache/prod/classes.php line 9300

solution add to routing.yml:
homepage:
    pattern: /homepage

понедельник, июля 30, 2012

Install cloudera cluster cheats, add VM node.

  1. disable selinux at all hosts /etc/selinux/config
  2. disable iptables at master or fix rules
  3. set all node hosts of cluster in all /etc/hosts
  4. change the host name in all /etc/cloudera-scm-agent/config.ini

  1. enter into runned instance and change mac adress in /etc/sysconfig/network-scripts/ifcfg-eth0
  2. shutdown
  3. change mac address into Virtual Box VM preferences
  4. start VM
  5. ifconfig
  6. add host to /etc/hosts
    • 10.0.0.10   node10.cloudera node10
  7. change host at  /etc/sysconfig/network
    • HOSTNAME=node10.clouder

вторник, июля 17, 2012

MapReduce doens't run when i tried install hadoop (CDH4) on Centos 6 (minimal) using Cloudera Manager (Free Edition)

I tried install full package include: HBase, HDFS, Hue, MapReduce, Oozie, ZooKeeper

But on start MapReduce I got error that JobTracker doesn't want to run.

After a few hours I found in log /var/log/hadoop-0.20-mapreduce/hadoop-cmf-mapreduce1-TASKTRACKER-localhost.localdomain.log.out this row:
 INFO org.apache.hadoop.mapred.TaskTracker: Tasktracker disallowed by JobTracker
solution found in google :
I added the following to the "MapReduce Service Configuration Safety
Valve" in the configuration of the MapReduce service page.

 
    mapred.hosts
    /root/mapred_approved_

hosts
    true
 

Then created /root/mapred_approved_hosts containing:
localhost
localhost.localdomain
In  Cloudera Manager click Services->MapReduce in category search type: "MapReduce Service Configuration Safety Valve" edit value and save, after it restart all services

p.s. so MapReduse run but i have problem to start Beeswax Server :)

Beeswax Server problem fixed, port 8002 used by other program

[me@localhost]# nmap -sT -O localhost
8002/tcp  open  teradataordbms

so I changed  port of Beeswax to 8004 and restarted service






вторник, июня 19, 2012

Memcacheq init.d script

source: memcacheq by Alexander Titov (qik) 
I removed few parameters from line 43
  • vim /etc/init.d/memcacheq
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
#! /bin/sh
#
# chkconfig: - 55 45
# description:  The memcacheq daemon is a network memory cache service.
# processname: memcacheq
# config: /etc/sysconfig/memcacheq
# pidfile: /var/run/memcached/memcacheq.pid

# Standard LSB functions
#. /lib/lsb/init-functions

# Source function library.
. /etc/init.d/functions

if [ -f /etc/sysconfig/memcacheq ];then
        . /etc/sysconfig/memcacheq
fi

# Check that networking is up.
. /etc/sysconfig/network

if [ "$NETWORKING" = "no" ]
then
        exit 0
fi

RETVAL=0
prog="memcacheq"

start () {
        echo -n $"Starting $prog: "

    if [ ! -e /var/run/memcacheq ]; then
        mkdir /var/run/memcacheq
    fi

        # insure that /var/run/memcacheq has proper permissions
    if [ "`stat -c %U /var/run/memcacheq`" != "$USER" ]; then
        chown $USER /var/run/memcacheq
    fi

        daemon --pidfile /var/run/memcacheq/memcacheq.pid /usr/local/bin/memcacheq -d -N -p $PORT -u $USER  -H "$DATADIR" -B $MAXMESSAGE -L $LOGBUF -P /var/run/memcacheq/memcacheq.pid $OPTIONS
#        daemon --pidfile /var/run/memcacheq/memcacheq.pid /usr/local/bin/memcacheq -d -N -p $PORT -u $USER  -m $CACHESIZE -c $MAXCONN -H "$DATADIR" -B $MAXMESSAGE -A $PAGESIZE -L $LOGBUF -P /var/run/memcacheq/memcacheq.pid $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch /var/lock/subsys/memcacheq
}
stop () {
        echo -n $"Stopping $prog: "
        killproc -p /var/run/memcacheq/memcacheq.pid /usr/local/bin/memcacheq
        RETVAL=$?
        echo
        if [ $RETVAL -eq 0 ] ; then
            rm -f /var/lock/subsys/memcacheq
            rm -f /var/run/memcacheq.pid
        fi
}

restart () {
        stop
        start
}


# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        status memcacheq
        ;;
  restart|reload|force-reload)
        restart
        ;;
  condrestart)
        [ -f /var/lock/subsys/memcacheq ] && restart || :
        ;;
  *)
        echo $"Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
        exit 1
esac

exit $?
  • vim /etc/sysconfig/memcacheq
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Configuration file for the memcacheq service.

DATADIR=/data1/memcacheq

LOGBUF=1024

MAXMESSAGE=1024

OPTIONS='> /data1/mq_error.log'

USER=root

PORT=22201
  • service memcacheq start
  • chkconfig memcacheq on

понедельник, июня 18, 2012

Create partition table and format disk (Centos 6) Softlayer

At SoftLayer i created server with two hard disks: primary (you can choose size max 100GB) and secondary (you can choose size max 300GB)
  • df doesn't show the secondary disk, you need to create partition, format and mount disk first
[root@server me]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda2           103108896  91059600   6811588  94% /
tmpfs                  1987292         0   1987292   0% /dev/shm
/dev/xvda1               99150     49491     44539  53% /boot
  • to see all disks run fdisk
[root@server me]# fdisk -l
Disk /dev/xvdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00025cdb

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1               1         261     2096451   82  Linux swap / Solaris

Disk /dev/xvdc: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdc doesn't contain a valid partition table

Disk /dev/xvda: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000cb5

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *           1          13      102400   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/xvda2              13       13055   104754176   83  Linux

primary disk is  /dev/xvda, and secondary is /dev/xvdc

run parted to create partition table, run mkfs to format

  • you will get into parted shell, run mklabel to set disk label (msdos for linux)
[root@server me]# parted /dev/xvdc mklabel msdos
Warning: The existing disk label on /dev/xvdc will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? Yes
Information: You may need to update /etc/fstab.
  • run mkpart to create partition start from position 0 until end of disk -1, for ext3 filesystem
[root@server me]# parted /dev/xvdc mkpart
Partition type?  primary/extended? primary
File system type?  [ext2]? ext3
Start? 0
End? -1
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? Ignore
Information: You may need to update /etc/fstab.
  • run fdisk to see partition num, now we see all disk normal at fdisk, but df still doesn't show secondary disk
[root@server me]# fdisk -l

Disk /dev/xvdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00025cdb

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1               1         261     2096451   82  Linux swap / Solaris

Disk /dev/xvdc: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003cf5d

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdc1               1       39163   314571823+  83  Linux

Disk /dev/xvda: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000cb5

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *           1          13      102400   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/xvda2              13       13055   104754176   83  Linux
[root@server me]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda2           103108896  91019892   6851296  93% /
tmpfs                  1987292         0   1987292   0% /dev/shm
/dev/xvda1               99150     49491     44539  53% /boot
  • format partition xvdc1 by mkfs
[root@server me]# mkfs.ext3 /dev/xvdc1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
19660800 inodes, 78642955 blocks
3932147 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
2400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 30 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
  • mount disk
[root@server me]# mkdir /mnt/xvdc1
[root@server me]# mount /dev/xvdc1 /mnt/xvdc1
  • add mounted disk to fstab
[root@server me]# vim /etc/fstab
....
/dev/xvdc1              /mnt/xvdc1              ext3    defaults        0 0
  • check
[root@server me]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda2           103108896  63571080  34300108  65% /
tmpfs                  1987292         0   1987292   0% /dev/shm
/dev/xvda1               99150     49491     44539  53% /boot
/dev/xvdc1           309636140    195548 293712004   1% /mnt/xvdc1

Полезно: Шифрование флешки в Linux

среда, июня 13, 2012

Mount additional disk to VirtualBox image (centos 6)

  1. shut down you virtual machine
  2. open VirtualBox Manager
    • choose you machine
    • click Settings->Storage->SATA Contoller->icon Add Hard Disk
    • click "Create new disk"
    • choose format VMDK (Virtual Machine Disk) for example -> next
    • choose Dynamically allocated for example -> next
    • enter name, and choose size
    • click Create
    • click Ok
  3. run virtual machine
  4. > sudo fdisk -l

    Disk /dev/sda: 5475 MB, 5475663872 bytes
    4 heads, 32 sectors/track, 83552 cylinders
    Units = cylinders of 128 * 512 = 65536 bytes

       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1               1       78125     4999984   83  Linux
    /dev/sda2           78126       79681       99584   83  Linux

    Disk /dev/sdb: 107.3 GB, 107374182400 bytes
    255 heads, 63 sectors/track, 13054 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Disk /dev/sdb doesn't contain a valid partition table
    • I added 100Gb disk that got name /dev/sdb
    • "Disk /dev/sdb doesn't contain a valid partition table" mean that you need to format this disk first
      • yum install gparted
      • run gparted (if you use graphical or see the manual)
      • choose you disk in right top corner dropdown
      • Device -> Create Partition Table -> choose ms-dos -> apply
      • right click on disk bar -> New -> +Add
      • right click on disk bar -> Format to.. -> choose ext3

  5. > mkdir /mnt/mydisk
  6. > mount -t ext3 /dev/sdb1 /mnt/mydisk
  7. > cd /mnt/mydisk
  8. > df 
  9. to mount automatically on boot, add disk to /etc/fstab my /etc/fstb for example, i added last line
    • LABEL=79d3d2d4  / ext4    defaults,noatime 0 0
      LABEL=e21a3e10  /boot ext3    defaults,noatime 0 0
      devpts     /dev/pts  devpts  gid=5,mode=620   0 0
      tmpfs      /dev/shm  tmpfs   defaults         0 0
      proc       /proc     proc    defaults         0 0
      sysfs      /sys      sysfs   defaults         0 0
      /dev/sdb1   /mnt/mydisk   ext3  defaults      0 0
  10. DONE

среда, мая 02, 2012

ERROR 1005 (HY000): Can't create table 'dbname.tablename' (errno: 121)

При попытке создать таблицу выдает ошибку:

 ERROR 1005 (HY000): Can't create table 'dbname.tablename' (errno: 121)

Имена foreign keys совпадали с именами foreign keys из другой таблицы. Переименовал все заработало.

среда, апреля 25, 2012

mysqld: unrecognized service


Centos 6 установил Percona server
yum install Percona-Server-server-55.x86_64

все поставилось
пытаюсь запустить через сервис, выкидывает ошибку
service mysqld start
mysqld: unrecognized service

в итоге скопировал /etc/rc.d/init.d/mysqld с работающего сервера с mysql

через неделю ставил слэйв и опять тоже самое, совсем забыл про то что было на прошлой неделе. в итоге через двадцать минут наткнулся на /etc/rc.d/init.d/mysql для запуска перконы
вот что значит сила привычки

service mysql start
MySQL: Starting MySQL….. ERROR! The server quit without updating PID file

в логе написано проверить что ничего не бежит на 3306
нашел mysqld, убил
все равно не запускается

после нескольких попыток 
service mysql start
Starting MySQL (Percona Server) SUCCESS!

что сделал не понял но заработало :)

среда, апреля 11, 2012

sphinxql: syntax error, unexpected IDENT, expecting CONST_INT or CONST_FLOAT or '-' near 'distance'

при попытке поиска со сравненим дистанции в индексе
SELECT id, weight() as real_weight, timestamp_created, GEODIST(0.55078861166627,0.59535898534089,lat_radians,long_radians) as distance FROM rt_matchWHERE distance <=115000.00 and max_distance<distance;
получаю ошибку:
sphinxql: syntax error, unexpected IDENT, expecting CONST_INT or CONST_FLOAT or '-' near 'distance'
в индексе таблица вида:
index rt_match { type = rt path = /var/lib/sphinx/rt_match rt_attr_timestamp = timestamp_created .... rt_attr_float = lat_radians rt_attr_float = long_radians rt_attr_float = max_distance .... docinfo = extern morphology = none min_word_len = 1 min_infix_len = 2 infix_fields = main_title html_strip = 1 charset_type = utf-8 enable_star = 1 rt_mem_limit = 256M }
кругом один флоат проблем недолжно быть попробовал другой вариант
SELECT id, weight() as real_weight, timestamp_created, GEODIST(0.55078861166627,0.59535898534089,lat_radians,long_radians) as distance FROM rt_user_match WHERE distance <=115000.00 and max_distance-distance&rt;0
получил ошибку:
sphinxql: syntax error, unexpected '-', expecting BETWEEN (or 8 other tokens) near '-distance&rt;0'
после снизошло озарение что sphinx не хочет высчитывать значение в where вынес вычисление в select
SELECT id, weight() as real_weight, timestamp_created, GEODIST(0.55078861166627,0.59535898534089,lat_radians,long_radians) as distance, max_distance-distance as hu FROM rt_user_match WHERE distance <=115000.00 and hu&rt;0.00;
снизошла благодать:
+------+-------------+-------------------+---------------+---------------+ | id | real_weight | timestamp_created | distance | hu | +------+-------------+-------------------+---------------+---------------+ | 2097 | 1 | 1334076081 | 101040.648438 | 386959.343750 | | 2103 | 1 | 1334076082 | 48350.027344 | 155649.968750 | | 2104 | 1 | 1334076082 | 92743.218750 | 334256.781250 | | 2106 | 1 | 1334076083 | 63040.796875 | 223959.203125 | +------+-------------+-------------------+---------------+---------------+

среда, марта 07, 2012

Подключение к SphinxQL, sphinx.conf с поддержкой иврита

гребаны SphinxQL оказвается чтоб подключится к нему из комманд лайн нужно указать тип соединения:
mysql --port=9306 --protocol=tcp

ну и для примера простенький sphinx.conf с поддержкой иврита

#
# Minimal Sphinx configuration sample (clean, simple, functional)
#

source job_title_source
{
        type                    = mysql

        sql_host                = localhost
        sql_user                = sphinx
        sql_pass                = PASS
        sql_db                  = DBNAME
        sql_port                = 3306  # optional, default is 3306

        sql_query   = \
                SELECT id, title, UNIX_TIMESTAMP(datetime_created) as timestamp_created, access_id, lang_id \
                FROM job_title \

        sql_attr_uint = lang_id
        sql_attr_timestamp  = timestamp_created
        sql_field_string = access_id

        sql_query_info          = SELECT * FROM job_title WHERE id=$id
}


index job_title_index
{
        source                  = job_title_source
        path                    = /var/lib/sphinx/job_title_index
        docinfo                 = extern
        charset_type = utf-8

        charset_table   = 0..9, A..Z->a..z, _-> , a..z, \
         U+FB1D->U+05D9, U+FB1F->U+05F2, U+FB20->U+05E2, U+FB21->U+05D0, U+FB22->U+05D3, U+FB23->U+05D4, U+FB24->U+05DB, U+FB25->U+05DC, U+FB26->U+05DD, U+FB27->U+05E8, U+FB28->U+05EA, U+FB2A->U+05E9, U+FB2B->U+05E9, U+FB2C->U+05E9, U+FB2D->U+05E9, U+FB2E->U+05D0, U+FB2F->U+05D0, U+FB30->U+05D0, U+FB31->U+05D1, U+FB32->U+05D2, U+FB33->U+05D3, U+FB34->U+05D4, U+FB35->U+05D5, U+FB36->U+05D6, U+FB38->U+05D8, U+FB39->U+05D9, U+FB3A->U+05DA, U+FB3B->U+05DB, U+FB3C->U+05DC, U+FB3E->U+05DE, U+FB40->U+05E0, U+FB41->U+05E1, U+FB43->U+05E3, U+FB44->U+05E4, U+FB46->U+05E6, U+FB47->U+05E7, U+FB48->U+05E8, U+FB49->U+05E9, U+FB4A->U+05EA, U+FB4B->U+05D5, U+FB4C->U+05D1, U+FB4D->U+05DB, U+FB4E->U+05E4, U+FB4F->U+05D0, U+05D0..U+05F2

 # STRIP HTML TAGS
        html_strip = 1

}


indexer
{
        mem_limit               = 128M
}


searchd
{
        listen                  = localhost:9312
        listen                  = localhost:9306:mysql41
        log                     = /var/log/sphinx/searchd.log
        query_log               = /var/log/sphinx/query.log
        read_timeout            = 5
        max_children            = 30
        pid_file                = /var/run/sphinx/searchd.pid
        max_matches             = 10000
        seamless_rotate         = 1
        preopen_indexes         = 1
        unlink_old              = 1
        workers                 = threads # for RT to work
  compat_sphinxql_magics = 0 # the future is now
}

понедельник, марта 05, 2012

Zend Route in ini file

конфигурация роутинга хранится в routes.ini
один из роутов ведет в редактирование джоба

job_edit.type = "Zend_Controller_Router_Route_Regex"
job_edit.route = "job/(\d+)/edit"
job_edit.defaults.module = "companies"
job_edit.defaults.controller = "job"
job_edit.defaults.action = "edit"
job_edit.map.1 = "jobId"

задача создать правильную ссылку используя
$this->url(array('jobId' => $job['id']),'job_edit', true);

получаем ошибку
An error occurred
Application error
Exception information:
Message: Cannot assemble. Reversed route is not specified.

надо добавить строчку
job_edit.reverse="job/%d/edit"