• Ворчание ягнят

    Никита и Рахим запустили сайт, где можно посетовать на дебильный интерфейс. Встречайте, Ворчание ягнят. Написан на Кложе.

    Дизайн не моя стихия, и писать в блог про каждую кривую кнопку мне кажется лишним. Это скорее для Твиттера, но я с ним не дружен. Так что все наболевшее шлю в приват Никите, а он обещал запостить.

    Подписывайтесь по РСС, там будет изредка мелькать и от моего имени.

    UPD стрим в процессе создания:

  • Фаерфокс

    Мне кажется, в недалеком будущем Фаерфокс умрет. Не полностью и не буквально, конечно. Просто сократит долю рынка настолько, что станет неликвидным. Ему присвоят звание “жемчужины опен-сорса”, гики будут стенать, причитать и сидеть на неподдерживаемой версии трехлетней давности, как это случилось с Оперой после перехода на Веб-кит. На поддержке останутся FF-only легаси-системы.

    Кто-то сказал, что браузеры – продукт массового потребления. В целом, они все работают хорошо и покрывают нужды рядового пользователя. Уже нет проблем с отрисовкой элементов. Основные усилия уходят на то, чтобы еще теснее интегрировать браузер с операционной системой и сервисами компаний.

    В этом плане все в порядке у Хрома, Сафари и Эджа (который Эксплорер). Гугл вливает в браузер колоссальные деньги, поскольку Хром – это окно во все сервисы: рекламу, почту, карты. Сафари связан с Маком и системой Эпла. Эдж, понятно, по самые уши интегрирован с Виндой, Бингом, Икс-боксом, офисным пакетом.

    Одного браузера недостаточно, нужны сервисы. Свои некоторые программы Гугл подкрепляет расширениями в Хроме. Так, при открытии Гугло-дока Хром молча ставит расширения Google Docs, Google Sheets и другие. Так они работают быстрее. А с чем интегрироваться Фаерфоксу?

    На рынке могут появиться разработки для снобов вроде Вивальди. Они обязательно снискают своего пользователя, но никогда не выйдут на массовый рынок. Потенциально новый браузер могут предложить только гиганты вроде Амазона или Фейсбука.

    Лично мои претензии к Фаерфоксу в том, что во-первых, он работает медленней Хрома. Пусть меньше загружены процессор и память, мое время дороже. Второе, Фаерфокс весь из себя для гиков. Это слегка умиляет, но в долгосрочной перспективе раздражает, потому что браузер нужен не только для работы, но и развлечения.

  • Реальность

    В плане профессии я знаю очень мало.

    На Си и Ассемблере писал только в институте, уже ничего не помню.

    Плаваю в указателях, памяти, куче. Код на C++ не понимаю.

    О нейросетях и машин-лернинге имею самые смутные представления.

    Про Биткоин читал только пару бульварных статей.

    Знаю три команды из Гита. Путаю, что куда мерджить.

    Накупил книжек, поставил пылиться на полку.

    Не напишу и пяти строчек на баше без интернетов.

    Не дочитал SICP. Хотел сделать все практические задания, слился.

    Кнут только в самых влажных мечтах.

    Компилятор или язык в жизни не осилю.

    Пилил свой проект, бросил.

    Пишу на Кложе, но в Джаве ни бум-бум. На Кложе потому, что язык редкий.

    Всю жизнь клепаю формочки к базе.

    Ненавижу JS, потому что не понимаю его, или наоборот.

    Емакс и Лисп чтобы быть не таким как все.

    Говорю по-английски с проблемами. Ссу провести вебинар на английском.

    Гуглю каждый HTML-тег. Копипащу со Стека.

    Умничаю в чате. Высмеиваю мессаджеры, сам же ими пользуюсь.

    Не знаю нужных хоткеев, не ставлю клевых программ.

    Не трекаю время, непродуктивен. Прокрастинирую.

    Разбираюсь только в Лиспе, Мейк-файлах и немного в Постгресе. Тем и живу.

  • Conditional Queries in Datomic

    Let’s discuss one thing related to Clojure and Datomic that might be a bit tricky especially if you have never faced it so far.

    Imagine you return a list of some entities to the client. It might be a list of orders, visits or whatever. Usually, when a user is exploring a long list of something, it’s a good idea to let them filter the list by some criteria: price, date, age etc. On the server side, you need to apply those filters conditionally. Say, when the date_from query string parameter is passed, you apply it to the query as well or live it untouched otherwise. The same for the rest of filters.

    Things become more tough when filters are applied to foreign entities. For example, an order references a user, and a user references a department. If the department_name parameter has been passed, you should join all the required tables and filter a proper field.

    Joining all the tables even if no filters were supplied is a wrong approach. Join operations are expensive and should never be performed in vain. A toolset that joins tables should take into account which ones has already been added into a query and never link them twice.

    Such systems that control the cases mentioned above are names ORMs. They are pretty complicated and full of ugly code and implicit hacks. But on the top level they behave quite friendly. Say, in Django (a major Python framework) I would perform something like that:

    query = models.Order.objects.all()
    department_name = request.query_string.get("department_name")
    if department_name:
        query = query.filter(user__department_name=department_name)
    

    Looks pretty neat. This automatically joins users and department tables under the hood using inner join SQL clause and puts department.name = ? condition into where section.

    People who work with such non-wide spreaded languages as Clojure usually do not use ORMs. But still, we need to build complicated queries. The community offers a handful of libraries (YeSQL, HoneySQL, Korma) where a query is being constructed within data structures: vectors, maps. I’ve been always against that approach. Before getting more experienced with Clojure, I felt uncomfortable constructing nested vectors like this one:

    [:select [:foo :bar]
     :from :test
     :where [(= :name "test")
             (when age-param
               (> :age age-param))]]
    

    The reason why I do not appreciate that is I cannot see the final query behind brackets and colons. It will definitely fail me once I need to express something like this (a fragment of a query from production):

    select
      foo.id,
      array_agg(foo.type_id) filter (where foo is not null) as type_ids,
      foo.name
    from foo_table as foo
    

    Modern libraries say it’s easy and fun to express your queries through data structures, but it is not, really. It becomes a challenge when applying multiple conditions to a data structure without seeing the final result.

    A good approach might be using a templating system. Say, HugSQL library allows to inject Clojure snippets into your SQL query. Those snippets are surrounded with standard SQL comments so they do not break syntax. There won’t be an error if you copy and paste such a Clojure-instrumented query into some RDBS administration tool.

    Here is an example of declaring such a query:

    -- :name get-oreders :?
    select o.*
    from oreders o
    /*~ (when (:department-name params) ~*/
    join user u on o.user_id = u.id
    join departments d on user.department_id = d.id
    where
        d.name = :department-name
    /*~ ) ~*/
    order by o.created_at;
    

    Than it compiles into a Clojure function:

    ;; no joins, no filters
    (get-oreders db)
    
    ;; causes joins and filtering
    (let [dep-name (-> request :params :department-name)]
      (get-oreders db {:department-name dep-name}))
    

    You may go further and include Selmer, a template library inspired by Django. It’s aimed at HTML rendering first but still may be used for any kind of documents including SQL.

    As I see it, a good templating system would be enough to generate SQL that fits your business logic.

    Now I’d like to discuss the same problem when using Datomic instead of classical RDBS solutions. All the tutorials that I have read do not cover a case then you need to apply several filters to a query. Suddenly, it may really become a problem. Let’s return to our example with orders. Once you don’t have any filters, the query looks simple:

    '[:find (pull ?o [*])
      :in $
      :where [?o :order/number]]
    

    But if you’ve got a department name you need to:

    1. inject a new parameter into :in section;
    2. inject additional clauses into :where section;
    3. prevent joining a user or a department entities twice if you need to filter by other department or user field.

    As we’ve seen, once you have a template system, you may render SQL queries as well. All you need is to write a query, test how does it behaves and then wrap some parts of it with special conditional tags.

    In Datomic, a query is usually a vector of symbols. Moreover, an immutable one. Thus, you cannot modify a query and adding something in the middle of it would be difficult. Surely you could wrap a query into an atom or track indexes where to inject a new item somehow but all of that would be a mess.

    What I propose is using a special kind of a query represented as a map with :find, :where and other keys. As the Datomic documentation says, when processing a query, every vector is turned into a map anyway. If we had a map, it would be easier to inject new items into it.

    To avoid wrapping a map with an atom or redefining it continuously inside let clause, there is a great form named cond->. It is a mix of both threading macro and cond clause. It takes an initial value and a bunch of predicate/update pairs. If a predicate form evaluates in true, an update form is fired using the standard threading macro. Thus, an update form should be either a function or a list where the second argument is missing and will be substituted with a value from a previous pair.

    What’s the most interesting about cond-> is unlike cond or case forms, its branches are evaluated continuously. Each update form takes a value that a previous form has produced. In other terms, an initial value goes through multiple updates without being saved in some temporary variable.

    In example below, I’ve got a data set that consists from user, location and visit entities. Both user and location are simple ones and store just dates, strings and so on. A visit is a bit more complex. It means that a user has visited a location and assigned a mark to it. Therefore, a visit references a user and a location entities as well.

    The goal is to get an average mark for a specific location. In addition, such a value might be filtered by user’s age, gender or location’s country name. Those parameters come from a query string and could be either totally skipped, passed partially or completely. I’ve got a function that accepts a location id and a map of optional parameters:

    (defn location-avg
      [location-id {:keys [fromDate
                           toDate
                           fromAge
                           toAge
                           gender]}]
    
    

    The initial Datomic query:

    (def query-initial
      '{:find [(avg ?mark) .]
        :with [?v]
        :in [$ ?location]
        :args []
        :where [[?v :location ?location]
                [?v :mark ?mark]]})
    

    Now, here is a long pipeline with comments:

    (cond-> query-initial
    
      ;; First, add two initial arguments: database instance and location reference.
      ;; This form will always be evaluated.
      true
      (update :args conj
              (get-db)                    ;; returns the DB instance
              [:location/id location-id]) ;; location reference
    
      ;; If either from- or to- date were passed, join the `visit` entity
      ;; and bind its `visited_at` attribute to the `?visited-at` variable.
      (or fromDate toDate)
      (update :where conj
              '[?v :visited_at ?visited-at])
    
      ;; If the `fromDate` filter was passed, do the following:
      ;; 1. add a parameter placeholder into the query;
      ;; 2. add an actual value to the arguments;
      ;; 3. add a proper condition against `?visited-at` variable
      ;; (remember, it was bound above).
      fromDate
      (->
       (update :in conj '?fromDate)
       (update :args conj fromDate)
       (update :where conj
               '[(> ?visited-at ?fromDate)]))
    
      ;; Do the same steps for the `toDate` filter,
      ;; but the condition slightly differs (< instead of >).
      toDate
      (->
       (update :in conj '?toDate)
       (update :args conj toDate)
       (update :where conj
               '[(< ?visited-at ?toDate)]))
    
      ;; To filter by user's fields, we bind a user reference
      ;; to the `?user` variable:
      (or fromAge toAge gender)
      (update :where conj
              '[?v :user ?user])
    
      ;; If from/to age filters we passed, bind user's age
      ;; to the `?birth-date` variable.
      (or fromAge toAge)
      (update :where conj
              '[?user :birth_date ?birth-date])
    
      ;; Then add placeholders, arguments and where clauses
      ;; for specific filters: fromAge, if passed...
      fromAge
      (->
       (update :in conj '?fromAge)
       (update :args conj (age-to-ts fromAge))
       (update :where conj
               '[(< ?birth-date ?fromAge)]))
    
      ;; ...and the same for toAge.
      toAge
      (->
       (update :in conj '?toAge)
       (update :args conj (age-to-ts toAge))
       (update :where conj
               '[(> ?birth-date ?toAge)]))
    
      ;; To filter by gender, bind user's gender to a variable
      ;; and add a clause:
      gender
      (->
       (update :in conj '?gender)
       (update :args conj gender)
       (update :where conj
               '[?user :gender ?gender]))
    
      ;; The final step is to remap a query (see below).
      true
      remap-query)
    

    Remapping a query is important because the initial data is a bit wrong. The proper structure for a map query looks as follows:

    {:query <query-map>
     :args [db location_id fromDate ...]}
    

    In my case, I believe it’s simpler to keep one-level map rather than deal with two levels (:query first, then :args). It would force me to use update-in instead if just update and write more code. Here is the remap-query function:

    (defn remap-query
      [{args :args :as m}]
      {:query (dissoc m :args)
       :args args})
    

    Finally, let’s check our results. If somebody passes all the filters, the query will look like:

    {:query
     {:find [(avg ?mark) .],
      :with [?v],
      :in [$ ?location ?fromDate ?toDate ?fromAge ?toAge ?gender],
      :where
      [[?v :location ?location]
       [?v :mark ?mark]
       [?v :visited_at ?visited-at]
       [(> ?visited-at ?fromDate)]
       [(< ?visited-at ?toDate)]
       [?v :user ?user]
       [?user :birth_date ?birth-date]
       [(< ?birth-date ?fromAge)]
       [(> ?birth-date ?toAge)]
       [?user :gender ?gender]]},
     :args [<db-object> [:location/id 42] 1504210734 1504280734 20 30 "m"]}
    

    Now you pass it into datomic.api/query function that accepts a map-like query. In my case, the result is something like: 4.18525.

    As you have seen, composing complicated queries with Datomic might a bit tricky due do immutability and differences between string templates and data structures. But still, Clojure provides rich set of tools for processing collections. In my case, the standard cond-> has made all the pipeline. Neither atoms nor other tricks to track the state were required. There is a common rule: once you’ve got stuck with a data structure, keep yourself from inventing “smart” ways to deal with it. There is probably a built-in macro in clojure.core for that.

  • Фейсбук прекрасен

    Каждый месяц повторяется печальное событие: мне нужно зайти в ФБ и сделать анонс митапа. Я тяну время, но часа X не избежать.

    facebook

    Даже не знаю, что может быть хуже Фейсбука: столь тормозной и неорганизованный сайт еще поискать. При том что денег у них до жопы.

    Погрязли в жадности и бюрократизме.

  • In-Memory SQLite Database In Clojure

    Recently, I’ve been working with a SQLite database using Clojure. In this post, I’d like to share my experience that I’ve got from that challenge.

    SQLite is a great tool used almost everywhere. Browsers and mobile devices use it a lot. A SQLite database is represented by a single file that makes it quite easy to share, backup and distribute. It supports most of the production-level databases’ features like triggers, recursive queries and so on.

    In addition, SQLite has a killer feature to be run completely in memory. So, instead of keeping an atom of nested maps, why not to store some temporary data into well organized tables?

    JDBC has SQLite support as well, you only need to install a driver, the documentation says. But then, I’ve got a problem dealing with in-memory database:

    (def spec
      {:classname   "org.sqlite.JDBC"
       :subprotocol "sqlite"
       :subname     ":memory:"})
    
    (jdbc/execute! spec "create table users (id integer)")
    (jdbc/query spec "select * from users")
    
    > SQLiteException [SQLITE_ERROR] SQL error or missing database
    > (no such table: users)  org.sqlite.core.DB.newSQLException (DB.java:909)
    

    What the… I’ve just created a table, why cannot you find it? An interesting note, if I set a proper file name for the :subname field, everything works fine. But I needed a in-memory database, not a file.

    After some hours of googling and reading the code I’ve found the solution.

    The thing is, JDBC does not track DB connections by default. Every time you call for (jdbc/*) function, you create a new connection, perform an operation and close it. For such persistent data storages like Postgres or MySQL that’ fine although not effective (in our project, we use HikariCP to have a pool of open connections).

    But for in-memory SQLite database, closing a connection to it leads to wiping the data completely out form the RAM. So you need to track the connection in more precision way. You will create a connection by yourself and close it when the work is done.

    First, let’s setup your project:

    :dependencies [...
                   [org.xerial/sqlite-jdbc "3.20.0"]
                   [org.clojure/java.jdbc "0.7.0"]
                   [com.layerware/hugsql "0.4.7"]
                   [mount "0.1.11"]
                   ...]
    

    and the database module:

    (ns project.db
      (:require [mount.core :as mount]
                [hugsql.core :as hugsql]
                [clojure.java.jdbc :as jdbc]))
    

    Declare a database URI as follows:

    (def db-uri "jdbc:sqlite::memory:"
    

    Our database shares two states: when it’s been set up and ready to work and when it has not. To keep the state, let’s use mount library:

    (declare db)
    
    (defn on-start []
      (let [spec {:connection-uri db-uri}
            conn (jdbc/get-connection spec)]
        (assoc spec :connection conn)))
    
    (defn on-stop []
      (-> db :connection .close)
      nil)
    
    (mount/defstate
      ^{:on-reload :noop}
      db
      :start (on-start)
      :stop (on-stop))
    

    Once you call (mount/start #'db), it becomes a map with the following fields:

    {:connection-uri "jdbc:sqlite::memory:"
     :connection <SomeJavaConnectionObject at 0x0...>}
    

    When any JDBC function or a method accepts that map, it checks for the :connection field. If it’s filled, JDBC uses that connection as well. If it’s not, a new connection is issued. In my case, every execute/query call created a new in-memory database and stopped it right after the call ends. That’s why the second query could not to find users table: because it was performed within another database.

    Now with the db started, you are welcome to perform all the standard jdbc operations:

    (jdbc/execute! db "create table users (id integer, name text)")
    (jdbc/insert! db :users {:id 1 :name "Ivan"})
    (jdbc/get-by-id db :users 1) ;; {:id 1 :name "Ivan"}
    (jdbc/find-by-keys db :users {:name "Ivan"}) ;; ({:id 1 :name "Ivan"})
    

    Finally, you stop the db calling (mount/stop #'db). The connection stops, the data disappears completely.

    For more complicated queries with joins, HugSQL library would be a good choice. Create a file queries.sql in your resources folder. Say, you want to write a complex query that filters a result by some values that probably are not set. Here is an example of what you should put into queries.sql file:

    -- :name get-user-visits :?
    select
        v.mark,
        v.visited_at,
        l.place
    from visits v
    join locations l on v.location = l.id
    where
        v.user = :user_id
        /*~ (when (:fromDate params) */
        and v.visited_at > :fromDate
        /*~ ) ~*/
        /*~ (when (:toDate params) */
        and v.visited_at < :toDate
        /*~ ) ~*/
        /*~ (when (:toDistance params) */
        and l.distance < :toDistance
        /*~ ) ~*/
        /*~ (when (:country params) */
        and l.country = :country
        /*~ ) ~*/
    order by
        v.visited_at;
    

    In your database module, put the following on the top level:

    (hugsql/def-db-fns "queries.sql")
    

    Now, every SQL template has become a plain Clojure function that takes a database and a map of additional parameters. To get all the visits in our application, we do:

    (get-user-visits db {:user_id 1 :fromDate 123456789 :country "SomePlace"})
    > ...gives a seq of maps...
    

    Hope this article will help those who’ve got stuck with SQLite.

  • Пять лет блогу

    То, что я давненько ничего не писал, связано с тем, что залип в конкурсе Highload Cup. Две недели ни о чем другом думать не мог, все пытался улучшить решение. И тут вспомнил, что у блога юбилей: первая запись была сделана пять лет назад, 8 августа 2012 года.

    Речь идет о наивной заметке под названием “Рассылка смс в Питоне”. Там я популярно рассказываю, как слать смс через апишку уже несуществующего сервиса. В те дни я работал в читинском Энергосбыте, писал на 1С, Дельфях и Питоне, а лиспы и ФП были влажными мечтами.

    Пять лет по меркам интернета все-таки срок. Много ребят, которых я читал, со временем забили на блоги, не продлили домены. А мне нравится: блог стал настоящим хобби, которое, надеюсь, продлится всю жизнь.

    За прошедшее время я написал 320 постов. Не так много, но я не гонюсь за количеством. Некоторые заметки, первоначально задуманные как пара абацев, разрастались до нескольких экранов.

    В один момент я поймал себя на мысли, что злоупотребляю низкими приемами: жалуюсь на сервисы, высмеиваю тех, кто не согласен со мной. Это работает, но не в том направлении, в котором бы мне хотелось. Теперь стараюсь писать только на важные темы: программирование, образование, карьера, книги.

    Было неприятно видеть, как любимые мной блоги вырождались в комментарии политических новостей. Эту тему я тоже решил пресечь.

    Решил двигаться в сторону англоязычной аудитории. В России на Кложе пишут три с половиной анонимуса, поэтому нет смысла писать о ней на русском.

    Технические заметки на английском порой дают сильный отклик. После публикации статьи о миграции с Постргреса на Датомик в первый же день ко мне зашли 9000 янглоязычных посетителей.

    После нескольких удачных статей про Кложу меня добавили в агрегатор Планета Кложи. Если в посте будет слово “Кложа” на английском, статья попадет в агрегатор. Поэтому если я не хочу, чтобы такое случилось, пишу на русском.

    В планах сделать на сайте теги, рубрикацию и прочие ништяки. Благо, блог статичный на Руби-движке, исходники на Гитхабе, и я знаю, у кого можно накопипастить. А пока вот просто список постов, заслуживающих внимания:

  • Conceptual languages

    This is a short note on what I’m thinking about programming languages.

    Briefly, I reckon some languages as being conceptual. In a conceptual language, every part of it obeys some global idea. This idea forms design of a language, the way it behaves and develops. If affects libraries, community and ecosystem.

    Take Lisp, for example. The idea is, when we store our code as data, it brings incredible possibilities to process code as data or turn the data into code. No other language may offer something like that, only Lisp does.

    In Erlang, every task is solved within a cascade of lightweight processes named actors. Two actors may communicate directly being run on different servers. Actors do not create Unix threads. They are managed by OTP rather than operation system.

    With Haskell, you’ve got quite strong and flexible type system. Types are the most important part of a typical Haskell program. Once it compiles, it will work for sure.

    Clojure, a modern Lisp dialect, provides immutable data structures and powerful abstractions for concurrency.

    Respectively, non-conceptual languages do not have such a global approach. They try to implement as many features as possible to satisfy every domain: OOP, anonymous functions (lambdas), lazy evaluation, etc. As a result, we’ve got everything but nothing: each part of such a language is not as powerful as its analogies form those ones I mentioned above.

    Take Python, for example. Although it has such basic functional blocks as map, reduce and lambdas, programming with it in a functional way would be a mess.

    Every part of classical Javascript is just ugly.

    Java, a language with static type system, allows you to pass Null instead of any object and end up with NPE exception.

    Although conceptual languages are not perfect, they seem to be easier for me to learn because they have some common rules that could not be broken. Say, in Haskell, you just do not have plain Null value. In Clojure, you cannot modify a dictionary, and so on.

    They cannot be substituted with other languages. Really, how can you substitute Lisp or Erlang? There aren’t any alternatives for them.

    I believe, the future is about conceptual languages. To develop AI or distributed systems, we need something more sensible than yet another language with classes and syntactic sugar.

    I’m not sure it could be Lisp, but something that borrows most of its features.

  • Weekly links #28

  • Weekly links #27

    I’ve just returned from EuroClojure, it was amazing! Thank you everybody who brought it to us.

    Here are some interesting links I found on Medium:

Страница 50 из 82