Planet Plone

Update your robots.txt

2013-06-19T13:45:38Z

We recently had an interesting afternoon when our servers were all but brought down by a crawler called 80legs. This crawler showed no mercy on our poor site and bore down with great vengeance and furious anger on articles so old that the cache didn't have them. We get a lot of crawlers, if you've spent hours gazing at your instance-Z2.logs like i have (beats a washing machine!), you've seen a

Hotfix Posted for security vulnerability 20130618

2013-06-18T17:55:00Z

Users of Plone 4.3, Plone 4.2, Plone 4.1, Plone 4, Plone 3, Plone 2.5 and Plone 2.1 should immediately apply this hotfix.

How to build Python, so that a shared library (libpythonX.Y.so) is produced.

2013-06-16T13:27:34Z

For an optimal configuration of the Apache module mod_wsgi, Graham Dumpleton recommends:

“you are highly encouraged to ensure that your Python installation has been configured and compiled with the ‘–enable-shared’ option to enable the production and use of a shared library for Python.”

(http://code.google.com/p/modwsgi/wiki/InstallationIssues : “Lack Of Python Shared Library”)

This recipe might save you a few minutes’ research:

wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz
tar vxzf Python-2.7.3.tgz
cd Python-2.7.3
PREFIX=$HOME/local/python-2.7.3
export LD_RUN_PATH=$PREFIX/lib
./configure --prefix=$PREFIX --enable-shared
make
make install

Another thing is that nginx might be a better choice for the “front office”. But anything with a red star in its logo simply makes us old farts suspicious…

PSM13 talk videos are online

2013-06-16T02:49:24Z

The videos of talks given at Plone Symposium Midwest 2013 are now available at

https://midwest.plonesymp.org/video/video-recording

The videos of keynotes will be posted early this coming week.

Plone: Future or Legacy?

2013-06-15T06:18:25Z

This is a personal followup on several discussions I had in personal lately with other Plone integrators and big Plone users in addition to Steve's and Eric's latest blog posts on the future of Plone (5):

Steve McMahon: The Plone 4 generation

Eric Steele: Plone 5

Most of us are in weird situation because we have multiple and different roles in the #Plone world: either as integrators, as contributors, as Plone users.

So where is my current problem?

Steve numbers clearly show a growing number of contributors. On the other hand there is the overall feeling that Plone turned into a legacy technology that will be around for many years but the overall impression is that the overall adoption of Plone is stagnating. This impression is of course subjective and based on personal experience and talks with customers. In addition many so-called Plone companies are currently making a shift by no longer focusing on Plone but also providing services in other areas and focusing on solution and services to customers and addressing their needs instead of advertising themselves as "Plone company". The concept of the "Plone company" is going to fail for most Plone shops. Customers are looking for solutions and as integrators we have to have various solution options - Plone will be one out of many others. The decade of the CMS systems is over. CMS turned into commodity software. They were something new and cool in the first decade of the millennium but nowadays web technology moved and the technology and requirements are more diverse than ever. So content management is now only a small fraction of "the web" as it was some years ago. I think the trend of stagnation and more slowly adoption is also true for other content management systems.

Going back of the level of adoption of Plone. The level of adoption of an enterprise-level software is possibly depending on factors like

marketing

user experience

to a certain degree: underlaying technology

Steve speaks in the last paragraph of his posting of a marketing problem or lack of Plone marketing in the past. I would object that this is has been a real problem and does not explain the stagnation of Plone. In Germany we found the German Zope User Group (DZUG) around 2002 and made annual conferences around Zope & Plone. The DZUG invested a lot of marketing work in promoting Plone through the participation in various fairs (like Linuxtag, Froscon etc.). There was certaintly an outcome by attracting some new people and institutions to Plone but we could actually never compete with our competitors like Joomala are the special German phenomeon Typo 3.

The more important reason for the stagnation of Plone comes from my point of view from the user experience with Plone. User experience breaks down (for me) into usability of the system and professionalism and reliability of the system. At the latest conference of the German universities using Plone we had a pretty long and controversial discussion about how a usable CMS should like editor. My conclusion from this discussion: people love or hate Plone in the same way as it is true from the usability point of view for Typo 3. Power users seem to prefer Typo 3 while more average editors prefer Plone!? Anyway...this is a never ending discussion and there is no solution to make a CMS user interface work for all editors. This will always a love-hate relationship. Another point that we have in our own hands is the reliability and the professionalism of the overall system. I personally came across several weird issues lately and bitched about them in several controversial postings on the Plone product developers list (you will find them if you are interested :-). The key point is quality and stability.

Some examples:

Webdav support of Plone under Windows is fragile and not usable for enterprise-level usage. Many weird workaround exists for getting rid of webdav locking problems.

Customer received a Plone 4.2 buildout with a policy and theme package. Everything was working...the customer himself installed a few add-ons on its own...suddenly the complaint that TinyMCE would no longer work...even after debugging for some hours I could not find the real problem...perhaps not a core-issue but some weird interaction of Plone with some of the add-ons. Bad user experience, pissed integrator experience.

OOTB installation of add-ons must work. E.g. a simple functionality like video support in Plone has to work without issues by installing collective.flowplayer (4.0). As an integrator I can handle issues and properly fix them. End users just become frustrated.

Upgrade story: upgrades in Plone 4.x became an unpredictable pita. With 4.x update we had to update various add-ons basically because of changed imports. So some add-on in some version was only running with one particular Plone 4.x version. Having to support multiple Plone 4.x versions ended up with try..except cascades or different release branches....not nice.

Migration story: I upgraded a Plone 4.0 site to Plone 4.2 based on my own Produce & Publish framework. Suddenly some image URLs (using plone.app.imaging) could no longer be resolved directly through restrictedTraverse() for whatever reason. In addition: Plone started to store image URLs (as edited through TinyMCE) suddenly with the full host and port which became a major pita in the context of virtual hosting and in particular in the context of having image URLs starting with https:// accessed through the public website served over http:// (IE security restrictions caused the images to appear broken). All these issues caused major user pain and trouble on our site. In the end the migration took full six working days (two days scheduled and paid, four unpaid working days on the integrator side).

I am not sure with conclusion I should draw personally when we talk about the future of Plone. There is of course a future for Plone in certain areas like large institutions as intranet or portal but also the requirements here are shifting into a different direction ("social media"....). hm.....Plone is legacy for me...but an important legacy, one solution out of many others, a solution that fits a particular use-case: enterprise-level content-management.

Modern Plone Best Practices (or How to be ready for Plone 5)

2013-06-15T03:05:03Z
This poster is meant to keep developers on the "golden path" it is based on tips from Eleddy's Everything you never wanted to know about Plone 5. I'm hoping it will steer developers (old and new) in the right direction. There are really 6 tips, but 5 for Plone 5 sounds so much better than 6 for Plone 5, so I sneakily combined 2 tips :). Update: Hi resolution English and Spanish versions

Water software and python

2013-06-15T00:37:00Z

Weird link I got from a colleague this evening: some Dutch ICT awards were awarded this week. The link is in Dutch, but trust me if I say that the two (to me) relevant awards were awarded to Python (as a fantastic computer language) and to hydrologic (a Dutch water+ict company, for their water software).

For me, the weirdness is three-step process:

Ok, an award for Python is nice! Almost all our (Nelen & Schuurmans) software is written in Python. And virtually all of it is open source. Just look at github: https://github.com/nens/ and especially https://github.com/lizardsystem/ . Lots of Python and Django.

The water software award goes to a closed source Microsoft shop.

We (Nelen & Schuurmans) are the perfect combination of water software and open source Python.

We're a water consultancy company. And a successful one, too. Lots of great projects. Most of them centered around something called Lizard (http://lizard.net for the mostly-Dutch business language stuff, http://lizard.org for the English IT stuff). Lots of partners working on and with it. It is a real platform with every partner putting in ideas and expertise. Most of the actual programming happens at our place, to be fair, at the moment. But... I've now personally been involved with already three partner companies that have been working with the Python code.

Ok, now what's the weird thing? Well, the things being said in the other company's award text are actually the things I'd use to describe the Lizard products :-) The core tenet of Lizard is "combining data" and that's literally what's in the award text. I have got the videos and screenshots to prove it. I can show you the lines in lizard-map's models.py that are the core of our information-combination structure.

Well, let them have fun. If you're looking for water-related software, Lizard is the one you want. Combining data, friendly visualization, big data, flood calculations, a lot. And... most of it is open source. And our software combines water with the other award winner: Python! Open source Python water software. That's much better than the current water software award winner: I can hardly find a screenshot on their webpage. And not a lot of specifics. We have got all our stuff on github and a bunch of demo videos on http://video.nelen-schuurmans.nl/.

I'm very happy working where I work now. Water + open source + python: it can hardly get better. I notice the same mindset in my colleagues: we're a big force that's pushing forwards and putting out lots of very good software. Including source code, screen shots and videos. To me, that's pretty different from working for a company without open source, with microsoft-only software and with hardly a screenshot on their website. (Perhaps weird that I'm hammering on the lack of screenshots, but I still haven't managed to see one of "their" websites in real life, so I'm personally (probably unfairly) thinking about it as brochure-ware instead of as a potential threat...)

Wanna work on great Python water software in Utrecht? Mail us (provided you speak Dutch, that's pretty much a necessity). A nice working location dead smack in the center of Utrecht next to the oude gracht. Especially nice now that the weather is good!

Plone 5

2013-06-14T22:05:00Z

So, as you’ve likely noticed, we spent a lot time of talking about Plone 5 during the recent Plone Symposium Midwest. I think we’ve reached the point where can really start prioritizing concepts for a major release. Obviously, any changes need to go through the Framework Team for final approval, but I’ve got a bully pulpit and I’m not afraid to use it.

Here’s how I’d like to see development focused over the near term:

Plone 5 ships with Dexterity as its default content type story.

Dexterity versions of plone.app.event and plone.app.collection

A better set of widgets (plone.app.widgets)

Products.Archetypes, Products.ATContentTypes, archetypes.schemaextender remain available as add-ons for older content

Full multilingual story available (plone.app.multilingual)

Plone 5’s UI gets an extensive overhaul.

Diazo becomes the default theming story

Fewer CSS files makes finding and overriding easier

No more !important in its CSS

Editing interface is separated from content for easier styling (plone.app.toolbar)

Accessible: WCAG- and ATAG-compliant

Deco.gs is replaced with a more commonly-used grid system, responsive

New folder contents UI with filtering and batch operations

Improved, tested widgets

TinyMCE gets upgraded to version 4.0, with a simplified integration making it easier for us to stay up-to-date

Plone 5 ships with an empty portal_skins.

Exists for add-ons

Most content moved to browser views, z3c.form

Login rewritten to use views, events. Simple, but pluggable for more complex use cases

Pay attention to proper XSRF protection for our most common functionality

Plone 5 has amazing test coverage.

Migrating scripts from portal_skins to browser views allows more unit testing

New JavaScript practices mean more unit tests for our interactive stuff

Plone 5 is faster.

Chameleon reduces rendering times by 30%

Date formatting is handled on the client side

Plone 5 is easier to learn.

plone.api covers most common development tasks.

Plone 5 eats its own dog food and sets an example for:
Views over skins

z3c.form

Dexterity

Diazo

JavaScript integration/development requires less knowledge of Plone

Theming requires less knowledge of Plone, less fighting with the default setup

And most importantly…
Plone 5 is achievable within the next 12 months.

There are, I’m sure, 30 other features that are in some stage of not-doneness, and while I’d like to see us do everything, we need to find some kind of scope. I think the above featureset gives us a nice range of features, a not-too-painful upgrade, and gives everyone plenty to do over the next year.

We’ve already begun approaching Plone companies to get buy in to the effort, asking for time, money or organizational support. Six Feet Up has pledged 5 hours per week of their QA team’s time to test our work. Netsight is organizing a sprint to rework the login. AMP is pledging 5 hours a week toward deprecating portal_skins. Wildcard is taking on the folder contents revamp. If your group has a particular need, now is the time to make it happen.

In addition, we have plans to use the general community excitement that comes along with a major release to build some momentum in our non-code teams. If we can move this forward, I think Plone 5 will be a big step up for both Plone the software and Plone the community.

Operation Flawless Login

2013-06-14T19:11:19Z

No booze, no caffeine, training,
bad wifi, jetlag, and 100+ users
on every type of device under
the sun who couldn't reliably
log onto the system. PAIN.
I was going to do the whole play it cool thing and wait 3 days for the next post but I am just so excited about this new "blogging" trend. I'm hoping to make it a Friday tradition (preceded by 2 cups of drip coffee and followed by a nap).

One thing I'm super excited to start working on as part of the migration away from cpy/cpt is redoing login. As far as Plone goes, you are likely in 1 or 2 camps:
Works fine for me!
Rot in hell you !@#$ old code
I suspect most of you are in camp 1, or just have no opinion whatsoever so I'm hoping I can convince you to jump over to camp 2. Most of the following was originally written with an intense amount of pain (see photo) so if it seems like I'm shouting at times its because I was: I probably qualified as temporarily insane. Here we go!

Ryan Foster (@dextermilo) recently proposed this beautiful, simple, login page for a client which has all css/js/etc embedded making it load very fast (1 template + logo). Tell me you don't want this login page HARD. I know I do.
Login as a First Impression
For sites that are completely blocked off by default, login is the first experience with a new system/site. For anyone who is invited to a system, its the same case. In fact, I would argue that a seamless login is the most important introduction to any web based system. It should be flawless. Absolutely flawless.

And this is where "we the Plone" fall flat on our face. A sampling of the issues, mostly focused on the ones dear to my heart at the moment:
The login, registration, password reset pages are too slow from relying on a framework that it doesn't need, causing more complications with frustrated users. We can't login 100 people at once even with load balancing. Trainers around the world know exactly what I'm talking about. How can you ramp up a system that people can't login to fast and accurately?
We are 100% not ready for mobile. There are some issues with random phones that we will never be able to address but we also need to catch up with HOW people use mobile devices. The best example is the shitty network experience. What happens when you click on a registration link and the network goes out before you can complete? You go back to the email invite and RE-CLICK. But wait… whats that? Plone only allows you to click once and the link expires? 1999 called and wants their user restrictions back.
The messaging/wording of invite and password reset emails is robotic and offers no indication to what the user just signed up for or were invited to. They get the email and have questions like "Wait, so how to I get to the home page?" and "What is my username?" and "Who is admin?". If you have signed up for basically any service at any point lately I'm sure you know what a great invite email feels like.
Testing? What's that?
Have you seen this code? I mean seriously (WARNING: NSFW) look at that shit. It's 10 years of whacked out random bug fixes that no one has any idea if its ok to rip out or not. Its 20+ different files spewed throughout the code in portal_skins style that requires a diagram the size of my kitchen table to print out. Bugs languish in the tracker for years because everyone is terrified to touch it. It is officially unmaintainable code. Don't get me wrong, it was great in its hay day. But I'm sorry login_form.pt, it's time for us to go our separate ways.
Customization. Ugh. Enough said.
There is no OOB way to block off a site.
and so much more… I don't have time to list them all. Have a favorite login gripe? Comment below!
Summary: It's a liability to Plone and more importantly my projects, and my job these days is to minimize liability. I'm coming for you login!

Battling the login monster one head at a time takes too long. Instead we cut out the heart, then roast the heads in olive oil for 8 hours, top with a dollop of creme freche and a sprinkle of truffle oil. Serve immediately with a spicy Triple.
I Love You but I've Chosen Rewrite
In good faith of open source complaining, I am determined to help fix it. In fact, Matt Hamilton has already agreed to help and get this handled and I secretly suspect that he has more deep down pain than I since he is taking off with it and already planning a quick weekend sprint to kick it off. Are you one a member of the login burned? Help us out!

Here is a brain dump of the things I personally want to accomplish with login rework:
Login gets dirt fast. It should handle 100 simultaneous logins on 1 site, 1 box, standard hardware. I want to say 300 but 100 is a good target and we can't have mass training sessions without this. I think a lot of the other options can help on this. I am tired of pointing to caching as the solution. There are a lot of transaction issues in this but we have to start somewhere. The skeletons oh the skeletons!
Stop basing on main template (this also applies to 404, 500, etc). If a site is closed off and resources were just recooked, thats 20+ resources the browser has to load and thats the immediate impression for the users. Get this down to less than 3 files (a login splash type thing, ideally 1 page with embedded resources). Ability to customize VERY easily (diazo?) but more importantly having a good default.
Mobile. Responsive, quick. Avoid redirects at all costs and where they are make it sane. I recently saw password reset on initial login fail on a variety of mobile devices. We are doing something wrong/old school/something.
Integrate iw.rejectanynmous to core.
Sane invite messages. Default system invite messages are horrid. They don't have site urls, nor indicate the username or who or what or… anything really. Ideally this message can be customized TTW (e.g. welcome to the site. this is our new intranet and I want you to use it like x,y,z your login is x the url is y and I am your new overlord)
Forgotten password. Oh god. Expire on click is a disaster, and the methodology is generally buggy.
Tests. Tests everywhere. I want robot going crazy too.
Security. Lots of it.
Login/reset/all forms rewritten as z3c. This allows us to extend and add things like site agreements.
Simplicity. There are too many options to be sane at the moment. We need to not even look at the code, and think "what do people REALLY need to configure in login" (especially that can't be done in PAS). We need to worry about our 80% use case first. I will be rabid about this. Login, login, login. And that's it. For now.
Rip out openid as it is. A simple pluggable solution is the answer to proper integration.
Default captcha for open registration.
Integrate some of the login api work that has been done.
And on my "after that is done" wish list:
Must reset password is weird and has never worked properly as I have wanted it implemented. The ability to expire passwords from some control panel has been on a wish list for many clients over the years. Time based expiration as well. That works, not that is a hack into the current whacky code base.
Tracking for site admins. Who hasn't logged in in 2 weeks? 1 month? Who has NEVER logged in? How can I make sure people are actually using the site? What kind of data do we need to capture to answer security compliance questions? We can't hide behind LDAP integrations for that forever. Also not first pass, but the ability to generate these reports must be considered.
In the category of "really dreamy some day if I can find the time":
Plone as oauth consumer. It's 2013 and "login with Facebook" isn't going away. *sigh*. I don't know that this should be in core but the ability to do setup oauth is HAWT.
Plone as an authentication provider.
We are ripping out portal_skins piecemeal so that we can get the new stuff out to integrators ahead of time to test then merge it back into core for the release. Login is its own package, plone.future.login and anyone is welcome to help the rewrite. The theme is release early, release often, simple, fast, extensible.

Easy enough right?

The Plone 4 Generation

2013-06-14T15:30:00Z

Ken Wasetis gave a great talk at Plone Symposium Midwest comparing Plone with current versions of Drupal. The talk — which Ken gave in a very "just the facts" way — was mostly about the different strengths of the applications, but Ken also talked some about differences between the projects. A couple of Ken's slides showed Ohloh.net charts comparing commit and participation rates.

Take a look at the Commits per Month chart for Plone:

This is a remarkable chart. Plone development activity has increased dramatically over the past three years, and is now averaging close to 1k commits per month. Poking around Ohloh I was unable to find anything like this for any comparable project. Most of the projects that we think of as competitive show the opposite: declining commit rates.

Contributors per month is also impressive:

This is how a community open-source project should look. Note that this chart is not directly comparable to projects that have become largely single-company rather than community organized. Plone has actively sought to increase direct contributors, relying on code review and our test-driven culture to maintain quality. Some projects instead keep commit rights down to a small circle of patch reviewers.

Take a look again at that Commits chart. The big shift up occurs at roughly the point where Plone 4.0 alpha 1 became available. If you look at commits, that's also when we began seeing a new generation of developers — many from countries that had never had Plone core developers before that point. If you look at the leaders for commits in the last 12 months, many of those names were not on the Plone 3.x list.

Also at Plone Symposium Midwest, Eric Steele gave a great talk on inter-generational transfer of information in open-source software projects. (The actual title was different, but that's a big part of what I remember.) I think the case can be made that Plone has had a very successful generational transfer in core and add-on development.

Why has that happened? And why does Plone's activity level look so much better than projects we think of as competitive? Part of that's leadership. Eric Steele's way of handling the release manager role makes it clear that the project is open. His technical skill at managing an astonishing number of modules keeps it working. But, while it's more amorphous and hard to measure, I think culture is a bigger part. Our culture prizes collaboration. Sprints are where that culture is passed on. I suspect that our more intimate sprints may play the most important part, particularly those that bring long-time developers side-by-side with new contributors. There's probably a dissertation topic in there for some aspiring Sociologist or Anthropologist.

My ramble here has been focussed on core development. I think we've been far less successful with marketing and communications. In particular, I think we had an unfortunate period where we tried to model those parts of the project on the communications of more corporate, hierarchical organizations. That's shifting. I see some dedication to making marketing and communications more like the rest of the project: explicitly collaborative and welcoming to new faces. It'll be fun to see how it works out!

Charts are from ohloh.net and include only core commits.

fourdigits.remotemembrane released

2013-06-14T14:52:10Z

Wait, what?

fourdigits.remotemembrane is a PAS plugin, that delegates to a membrane PAS plugin in another Plone site within the same Zope instance.

Usecase

We have a group of four Plone sites in which one of them contains all the membrane objects. We needed all of the Plone sites to do authentication and authorisation on those membrane objects.

The PAS plugin implents the following API:
RemoteMembraneUserManager(MembraneUserManager)
:: IAuthenticationPlugin
:: IUserEnumerationPlugin
:: IUserIntrospection
:: IUserManagement
:: IMembraneUserChanger

RemoteMembraneUserFactory(MembraneUserFactory)
:: IMembraneUserFactoryPlugin

RemoteMembraneRoleManager(MembraneRoleManager)
:: IMembraneRoleManagerPlugin
:: IRolesPlugin

RemoteMembranePropertyManager(MembranePropertyManager)
:: IPropertiesPlugin
:: IMutablePropertiesPlugin

RemoteMembraneGroupManager(MembraneGroupManager)
:: IMembraneGroupManagerPlugin
:: IGroupEnumerationPlugin
:: IGroupsPlugin

Usage

Just add the plugins you need as normal plugins in acl_users and fill in the id of the remote Plone site.

Download / forks

We're on bitbucket: https://bitbucket.org/fourdigits/fourdigits.remotemembrane .
PyPI: https://pypi.python.org/pypi/fourdigits.remotemembrane/

Please, Plone developers, refrain from

2013-06-14T00:13:00Z

Hi people,

last week an absolutely fabulous gathering happened: Plone Symposium Midwest, in Oshkosh, Wisconsin, USA.

Much happened there, which you will read about in various other places. Especially, we also did some good work on accessibility. I’ll post much more on this in the coming weeks; stay posted.

One thing I want to stress today to the Plone community: we have been using the HTML tags of
and
for a lot of stuff. Most notably portlets.

This is wrong. Let me repeat this: this is wrong. You should only be using
and consorts if it is actually semantically correct. That means: you can use it if it is actually a definition list. You should not be using this for display purposes. Like, really not. No. Just don’t. Bunnies and kittens will die in vain when you do this. We love bunnies and kittens, and they should thrive. Trust me on this one.

We will change the whole portlet templates in the future. As many people are basing their own templates off of the standard ones, we won’t do this in the Plone 4.x series, so we won’t break your stuff unexpectedly, but we will do it in the future.

So please. If you want to future-proof your portlets: use something else. Use a header tag for the header, and use

for the content. Tack classes onto it so the design people can make it beautiful.

Just don’t use
and friends, unless your portlet is actually, semantically, really, seriously, and I mean this, an explanation of a term.

If you’re using it just to tack some CSS onto it, change your ways. Please, pretty please, with whipped cream on top. It will make our compliance with accessibility rules around the globe much easier.

Thank you.

Paul

PSM 2013 Days 1-4: Everything you Never Wanted to Know About Plone 5

2013-06-13T22:19:39Z

My, my, my it's been a while Ploners. I deeply apologize for lack of communication on my part and my only excuse is laziness, impatience, and hubris.

How does one sum up the realization that communication in the Plone community has hit rock bottom? One can't. So this will likely be the longest blog post I ever write and hopefully the last one of this length. Anyone who has ever said they get pissed that "we" don't communicate will leave this post with sincere regrets about asking for more info. If you are lucky I will put a TOC on this manifesto, but don't hold your breath. It's technical, it's marketing, and everything in between. Grab a cup of coffee and join me on the 7 day adventure that was Plone Symposium Midwest 2013 (PSM13).

Technical note: unlike certain content management systems like Plone, blogger does not have pull quotes. I'll do my best to mimic them for you lovelies, but I'm limited here.
Pre Sprint

Pre-sprinters at the park
PSM13 was kicked off by a pre-sprint. 10+ of us rolled into Oshkosh early, with the intention of focusing on the portal_skins removal plip. I arrived a day later than the usual suspects to see that things were already kicked off on redefining the javascript integration story for Plone which generally distracted us from the goal of coding for and drove most of us to the overarching question of... Seriously, can we define Plone 5 for real and really mean it?

Plotting, coffee, arguments, persuasion, a whole lot of momentum, and 2 days later, a document was born. That's right folks, its in writing, and its happening. Get ready.
Alanis Morisette InterludeThis may seem like a very trivial document folks, but I assure you its one of a kind. Not only does it list what we plan to do, but it even has marketing reasons and measures of completeness. Get your lottery ticket now. Book a flight to loch ness and prepare to capture sasquatch. This could never happen again.
I'm sure you don't trust us, and considering that the first talk of Plone 5 like things started in December of 2007, I totally understand. Why is this time different? First, all of these features are already in progress, and in fact many of them are complete. There was talk of a 4.4 release and that is likely hitting the wayside in exchange for the opportunity to make many overdue, much needed improvements (read:change is coming).
This Time, in the Key of A MajorWhat is a major release? A major release is an opportunity for us as developers to break some backwards compatibility in exchange for overdue large infrastructure upgrades and extra fancy new features. These changes affect highly customized sites the most so if you are running stock Plone, chances are that you are going to be A-OK. In general, we do our best not to break anything for ANY upgrades but some changes can't be avoided. Hence - major release!
Second, we reached out to integrators at the conference, business people, developers, etc to get feedback. What you see documented is by no means final, but rather a general consensus of what generally sounds reasonable to cut a major release. Think something is missing? Submit a PLIP NOW. Not sure why something is or isn't there? Ask a question in the comments and I will gladly explain. The posted document has open comments and if you want to have access to help us gather collective business reasoning, just ask.
Deco vs Collective.Cover or How I learned to Stop Hating and Start WAITINGOh dear. This was very much debated, but you'll notice these are both omitted from Plone 5. If we were to include either one of them, there is a good chance the release would be delayed even more, and both technologies have some serious technical issues before they can really be integrated. In addition, it was crossing the line of "maybe too much at once". They can still both be used as ad ons and I encourage the continuing development of both.
Third, we realized that the basic changes present a need to prepare add-on developers and integrators for the set of changes that are coming up. We need to get the community on best practices NOW so that the shock of Plone 5 doesn't hit us later. Almost all of the technologies and framework to make your custom integrations and add-ons compatible with Plone 5 are available for use TODAY. However, modern best practices are not documented well, and we realize that the success of Plone 5 relies on people upgrading their Plone 2.5 habits to modern development and integration practices. Documenting these best practices is at the top of all of our lists and we will do what we can to make sure that you aren't shocked.
Tough Love InterludeI know integrators, it's hard to change old habits. But, progress demands it. If you are particularly concerned about a change to the infrastructure and how it will affect your deploys, we will be putting together a FAQ to go with the release. Start asking here or on the PLIPs or on the Plone 5 document. We'll make sure that you're "story" has a translation. We aren't ask you to go back and upgrade all sites, just that you become familiar with modern Plone practices and start off new sites with them. A quick list:
Theme with Diazo
Use z3c.form for forms (not formlib)
Don't make CPT workflows (not like you wanted to anyways)
Use BrowserViews not portal_skins. Once you start, the fun won't stop.
Stop making custom archetypes and use Dexterity
Think about your reliance on z3c.jbot: in theory that is a task for Diazo these days. UPDATE: People are bitching about this statement. I don't know what to tell you. I prefixed it with "in theory" and a little "Think about" because I have never seen a need for jbot myself. That's right, I have never used it. I'm sure it's great, but my general thought is that it is a crutch for a more real problem that needs to be solved. This is not part of my expertise or something I care to fight about or elaborate on so here is my advice: do whatever you want. This blog post is not here to prevent you from thinking.
Fourth, we have no BDFL or dominant company at the moment to drive development and set a roadmap (and we are excited about it). Many historical leaders and companies have moved on to different places in life. All of the statistics show the community moving to a less centralized development pattern and at a faster rate (more on that research in a later blog post). When you see the data, you'll get tingles I'm sure. This means that we need to coordinate differently and make decisions differently.

Because of this, we realize the need to reach out for help to hit this vision so here is your ample notice and opportunity for all of the community to help us reach this goal. We aren't the community that can lock Martin Aspeli in a room and expect Deco to pop out a week later anymore. We have to be a bit more deliberate and communicate much more clearly. Someone said "failing to plan is planning to fail" and I almost spit out my beer. Fuck that shit, I'm planning to kick ass.

In particular, we are reaching out to companies to help us support this vision by offering resources to help out a particular PLIP. For example, SixFeetUp is offering QA time for new features like plone.app.events and Netsight is interested in helping get the Plone login story fixed (with one of my clients also having a vested interest). Please check out the story and ask your boss, clients, etc if there is some way that your company, institution, whatever can get behind a change and help is make Plone 5 better. I'll update this post with anyone who pledges to chip in!
We Need Marketers, not DevelopersOne thing that you will hear more of as these blog posts start pouring out of my butt is the notion that we aren't really suffering from lack of developers. We can always use more of course, but what we REALLY need are people with "soft skills". We need people who can start getting ready to market and push, who can make campaigns, and translate developer speak to business leaders and decision makers. Got some of those lying around? Contact me and we'll take advantage of them tomorrow.
In summary, Plone 5 is happening this time because we now understand the importance of communicating with all of you NOW and not after. All of us have promised to blog, talk, and communicate more. In exchange, we are asking for your help. Help as developers, integrators, and most importantly, business people and marketers. Key contacts are listed on the document and hopefully each leader will write a post explaining the change and what it means to you. When in doubt, just ask them.

Which is a perfect segue into my ~~personal vendetta~~ Plone 5 project: removal of portal_skins.
Goodbye CPY
Wildcards fantastic home brew
At the Plone Conference in 2012, I started working on the removal of cpy/vpy scripts. It's been slow and I confess I haven't dedicated the time I wanted. At this moment I think its for the better because there have been some realizations in the meanwhile on the right way to do this.

Initially I was focused on only upgrading cpy/vpy scripts but have broadened the scope to idealism in the way that only a masturbatory developer could: Down with portal_skins!

Now before you freak out, let me say that YOU, integrator, developer, all around good guy can still upgrade your plone product which uses things in portal_skins. However, my vision has a portal_skins free Products.CMFPlone. Everything that is there will be moved into plone_deprecated and you can let acquisition be all magical and do weird stuff still. In addition, your custom skins and what not will still work (there is no guarantee that this will still be the case in Plone NextBigRelease though). However, a major goal of Plone 5 is to return core to represent the current best practice of developing in Plone.

And now the most important question: WHY? portal_skins are part of this notion of acquisition. Security and performance reasons have show us that maybe this isn't the best way forward. In addition, modernizing the underlying framework relies on us moving beyond this notion. But wait, there's more. Most of that stuff existed before test cases were culturally enforced. The code is untested and intricate, so people are terrified to touch it. We can't fix bugs on files that everyone is afraid to modify. It get's better. Customization of page templates in portal_skins is shotty at best. It doesn't survive upgrades. Moving all forms to z3c.form gets us on one form technology that can be plugged and extended, and should survive future upgrades much better. Last but not least, times have changed. Resources like js and css shouldn't be managed in the registry. Anything bigger than a small site should be compressing with modern tools and hosting in a web server and/or a CDN. We need to recognize that the industry standards have changed and we are behind. Time to play a little catch up.

Moving all of portal_skins is a much larger task and this meant changing the approach to the change. At first it was a branch on Products.CMFPlone but I had a couple of realizations that led me to believe that the majority of changes should actually be done in a new package, which will then be merged back into core at the end. I'll bullet point the reasons since I'm running out of transition words:
Constantly merging back in changes in every package made me want to throw my laptop out of the window. Git has been merging like a dickhead lately and I had several errors from that waste a lot of time.
I want these changes for my own projects. Yesterday. I am willing and have the capacity to put them in live sites. Mostly performance reasons, but also integration pain points. I suspect other people do as well.
We can document and keep track of the effort in a more sane matter. When merging in commits I was losing track of the W5H of what happened because this is such a long ranging project.
Integrators and add on developers can pull the changes at any time and see if their shizzle works.
We have a lot of test cases to write. I'm emotionally tied to this thought that we can start fresh and outside of core will just seem more approachable.
All changes will go into plone.future.* packages that will be grouped by general integration points.

Grace Lim, the Plone Ranger
People will be able to grab them at any point. At the moment I see these as:
plone.future.formscripts: All of our core actions like a add, move, delete, etc.. are on code that hasn't been touched in YEARS. These are things that are highly unlikely to be uncustomized. This also includes control panel forms and their conversion to z3c forms. I can't imagine that most people can't start using this as soon as a release is cut. At first it will be just a few scripts and I will cut releases after each new thing is added. Ideally we can get these out in the wild, slowly, before the big bang.
plone.future.login: Oh god, login is worth a whole blogpost at least this length. It can't be approached in the same set of changes with any sort of sanity.
plone.future.resources: Plone 5 will have a new diazo theme so this won't affect you. Old themes will still work, but if you have properly based your site on Sunburst, then you have the OPTION to use this package, which will contain 1 properly minified js file, 1 properly minified css, and various icons moved out of portal_skins and into resource registries. I know I have several sites that likely won't redesign for a while and will benefit immensely from this.
In order to smooth this transition and to allow you as an add on developer and/or integrator there are a few other things that I intend to include:
A script to scan a directory of code that issues warnings for usages of things that have been deprecated.
A script to scan a directory of code that issue warnings for all plone.future references once the packages have all been merged back into core.
An intense upgrade guide that matches said warnings and indicates old school vs new school ways of doing things.
Tests, tests, tests!
Ideally each one of these items is updated AS the feature is removed to limit the last minute documentation work. I'm in the process of using cut, copy, paste, delete, and rename as an example and setting up this framework, after which I'll start cherry picking work that has already been done and calling for more help. The hard work and dedication that went into plone.api has been inspiring and its my hope to walk away from what could be a tedious task with an immense amount of pride. This is a chance to fix a whole helluva lot of things.

Overall, a pre-sprint was the best thing to happen to a symposium/conference since reliable wifi.
Fundraising
When you thought this blog post couldn't get any longer, it did. It will continue to grow until you all REGRET knowing what's going on. Should you have made it to this point, obviously you haven't learned.

It's like the city of Oshkosh just knew...
Being on the Plone Foundation board has been fruitful this year, and I'm super ecstatic about the push to spend money on sprints and kickstart this Plone 5 process. I was so excited in fact, that I gave a talk about it. With that victory has come a little burden to bear, and that burden is a bear by the name of "fundraising".

In order to dump money into sprints this year, I promised to raise the money we dumped for next year, so we can do the same thing. Turns out that I am a TERRIBLE fundraiser with a serious time deficit and intense lack of skills. Enter Jason Lantz, who gave a talk on how he built a fundraising platform on Plone for the innocence project. How disgustingly appropriate! After spending a full day hashing out ideas with him I realized how little I knew about raising money for non-profits and how much beer I will owe him the next time we meet up.

He has been a community lurker for over 10 years and is very excited to help us, the good people of Plone, get our fundraising shit together. His work is open and he is working on repackaging the add on for use by everyone (if you are a non-profit and you are interested, head over to github to see the progress). I am excited to work with Jason and in addition a whole new team of people excited to kick off fundraising efforts. Expect to hear more from us on that later and remember: you don't have to wait for a fancy donation website to support the foundation. Go to plone.org to help out now. The money that you donate is going to things like bringing K.K. Dhanesh over to Oshkosh from India to sprint for the first time!!
Communications
I've hinted at the severe lack of communication that goes on in Plone and we have lots of people (including myself) that have vowed to make this a little easier on people. This includes but is not limited too:
Eric Steele, Rok Garbas, David Glick, and myself vowing to blog more. Twitter has killed the blog and we need to go back to putting complete thoughts and ideas on paper for those that can't explain MultiAdapters in under 140 characters. I encourage all developers to get back to the paragraph.
Eric Steele will be trying to get together a regular call of team leaders, which will be attended by Rose Pryune who will act as a translator and post back to the community.
A revamp of the marketing team to be led by Eric Rozeboom. There is a lot to be done here and I'm sure he will post a wonderful plan soon and start recruiting. I know in particular that I am looking for someone to help us better utilize our adwords account. Know someone???
Heidi Reinke of UW Oshkosh College of Business has volunteered to have her marketing class for fall semester take on Plone as a use case for marketing. I can't wait to see what she comes up with and if you get contacted out of the blue by people doing research - don't be shocked!
And look guys, its in writing. No backing out now!

Lots of talk about communication led to some solid realizations on all sides. Developers don't like to communicate in lengthy posts like this and a lot of "translation" gets lost because of it. Forcing people to write docs is, however, not the solution. The oral interview is MUCH more effective and we have to spread that word. Want to know what's going on in Plone? Don't wait for a blog post. Ping a core member, ask them on a call, and write up a post for everyone else to consume.
That Stinking Contributor Agreement
In addition to being a Plone Foundation Sponsor, hosting a kick ass pre-sprint barbecue complete with home brew, and maintaining several high quality Plone plugins, Talin Senner of Wildcard Corp has volunteered to help us finally kick this long standing issue with the Plone Contributor Agreement. I am eternally grateful for them picking this up and rolling with it since it was making my head spin and I was dangerously spread thin.
Mailing Lists
We REALLY need someone to put a nice front end on mailing lists. The lovely Heidi made a statement along the lines of "why would i use plone? signing up for the mailing list makes this software look like its from 1990". OUCH. Interested? Ping me. We just need a plone.org initial UI into the basic mailing lists.

In addition, we should think about making plone-accounce a mail chimp list, considering that:

The UI doesn't suck at all
It's for communicating out, not within
I'm Running out of Steam
I didn't even get to talk about Grace Lims video project for coders and their kids, the Plone Ranger, the packages that were released and the intense code sessions, the work on EVERYTHING from coredev docks to mock, realization that we need help with adwords... oh god. That's another 6 hour plane ride of info. For now, sleep.

More sprint reports at:
http://www.uwosh.edu/ploneprojects/plone-activities-blog/
Followup Awesomeness:
http://pigeonflight.blogspot.com/2013/06/modern-plone-best-practices-or-how-to.html
http://pigeonflight.blogspot.com/2013/06/modern-plone-practices-poster-now.html

Communications

2013-06-13T19:43:55Z

“Every step is a first step if its a step in the right direction.”
- Terry Pratchett, “I Shall Wear Midnight”

I tend to be…measured in what I say. In the best case, that means that it takes me far longer than it should to communicate things. In the worst, it means that things just sit in my drafts folder.

Discussions during last week’s Plone Symposium Midwest point to many great things happening in the next 12 months. For me, the greatest is the rebooting of Plone’s communications team. Since last September, Rose Pruyne has been writing some fantastic articles for plone.org about community members doing great things with Plone. Now, she and a team of writers and editors will be conducting interviews with team leaders, developers, and anyone working to make Plone even better.

We recognize that we’ve been pretty lax about letting everyone know what’s going on, and this is our first step in fixing that.

Collective.twitterportlet and Twitter API version 1.1

2013-06-13T12:46:00Z

This week, on June 11th, Twitter retired version 1 of their API. As a result, the Twitter portlets of some of our customers stopped working. They are all using collective.twitterportlet so we created a quick (and slightly dirty?) fix to get them up and running again: edition1.twitterportletfix.

The problem

The obvious symptom we were confronted with, was that there were no tweets shown in the portlet. Instead, it contained the following text:

There was an error while rendering the portlet.

And the error log included entries like this:

AttributeError: 'unicode' object has no attribute 'get'

After vaguely recalling something about a Twitter API change, we started looking around and found the Twitter blog entry where they stated that the API v1 is retired and we should use API v1.1. But API version 1.1 requires you to use OAuth.

The good news is that collective.twitterportlet uses python-twitter. And that package is compatible with version 1.1 of the Twitter API. So upgrading to version (at least) version 1.0 of that package should at least make the Twitter API wrapper compatible.

The bad news is that just upgrading python-twitter is not enough. You still need to give it some keys (consumer key, consumer secret, et cetera) to get data from Twitter. Otherwise, you’ll see these kind of messages in you logs:

TwitterError: [{u'message': u'Bad Authentication data', u'code': 215}]

The optimal solution

In my humble opinion the optimal solution would be to do something similar as the combination of collective.facebook.portlets and collective.facebook.accounts provides. That is: allow the user to configure accounts (and perhaps even applications?) and let the user choose which account to use per portlet.

Unfortunately this involves more work and time than we could spend at the moment with customers waiting for their home page to look good again.

Update (2013-06-14): collective.twitter.accounts probably provides everything we’d need. Thanks for the tip Héctor!

The quick workaround

So instead of doing it the proper way, we decided to create a small package, edition1.twitterportletfix, that would solve our immediate need. What this package does is it allows the user to configure the required keys/tokens. It also customises the Renderer class of the portlet to use those keys to call the Twitter API.

Obviously this package will not fulfil all needs (and I’m not really proud of it) but for our use cases it should be enough for now. At least until there is a better solution available.

Why I chose this solution

(This section was added on 2013-06-14 after the question why it was easier to create this package than contribute to collective.twitterportlet. A very fair question.)

As I already stated, this is not an optimal solution and I’m not really proud of it. So why did I still go ahead with it? There are several reasons.

First of all, even if I would have liked to contribute, the PyPI page of collective.twitterportlet does not list a publicly available repository. Looking around on the collective repo on GitHub also did not turn up anything. I did find a repo with the same name but saw no relation between the owner of the repo and the owner of the package.

Furthermore, even if I had found a repository, contributing to the original package would have delayed this fix. My current solution does not provide a fix for every use case of the package. For instance, you cannot have two portlets showing different accounts. But building that proper solution (and effectively replicating the functionality of collective.twitter.accounts) would have taken more time and I needed a fix for our customers sooner rather than later.

(My focus was on fixing collective.twitterportlet rather than replacing it. As a result I failed to look around more and did not know about collective.twitter.accounts until Héctor commented on this article. Otherwise I would have tried to have collective.twitterportlet use collective.twitter.accounts, instead of reinventing the wheel.)

So I was fully aware that edition1.twitterportletfix is not a permanent solution. That is why I tried to make sure that you can cleanly uninstall the package. Once there is a better solution available—or you decide to replace collective.twitterportlet completely—you should be able to uninstall the fix and not leave a trace.