Coming back online…

Posted · 143 Comments
coming-back

For those who are not aware, PageLines.com’s static site files were deleted from the system not once, but twice on launch day.  We’ve been working at getting everything back to where it’s supposed to be; and I’ll write an update here.

Apologies

First, just wanted to apologize for these issues. We’re not as responsive as we usually are, and we really REALLY would like to get back to creating AMAZING experiences for PageLines users.

The Systems We’re Working On…

Here are the main system that are down and the timeline to getting things back…

  • DMS Pro Activations:
    Some users are experiencing issues with activations. To our knowledge 80% of these should be fixed now. Any lingering issues will be dealt with by end of weekend.
  • Forums:
    The PageLines forums are going to be moved to: http://forum.pagelines.com and should be back up by tomorrow morning. 
  • Legacy “Launchpad” Customer System:
    This will be moved to http://launchpad.pagelines.com and should be back up for you by tomorrow morning.
  • Upgrading Grandfathered Plus Subscribers:
    Working on this as well, everyone who was on Plus will be grandfathered to DMS Pro Business or Developer by Monday.
Update: July 30th

By End of Week:

  • Fix on v2 Store API; which will repair v2 store, other issues.
  • v2 Store System,
  • Finishing up grandfathering Plus members into DMS Pro

The Story

First Hack

At around 11pm PST on Tuesday, as we were preparing to launch, someone noticed that PageLines.com was down.

When we went into the system to figure it out something was definitely wrong… Somehow all static files on the server had been deleted.

At this point, we suspected a hack but weren’t sure. PageLines has a great backup system and so the process of restoring the site began.  By about midnight PST the team had reoriented itself and discovered that some scripts and files were lost and would need to be rewritten.

By 4am PST the site had been fully restored to launch-ready status.

Second Hack

The team began to consider the possibilities as to what had happened and decided that *if* the server had been compromised, restoring a back up would not fully resolve the issue as that vulnerability would be in the backup. But it was the fastest way to get back up and running.

However, we did suspect that if it *was* a hack, the hacker would not be satisfied with how fast we’d recovered and would do it again in the morning.

Sure enough…

With the official Launch scheduled for 10:30am, at 10:21am, the site was compromised and all static files were lost for a second time.

Time for a new server

At this point, no choice but to switch servers. So I called my friends at ZippyKid and FlyWheel; both wanted to help.

The server switch is where the issues come in, since everything has to be reconfigured and reinstalled. 

It’s important to note here that PageLines takes security very seriously.  Especially the security of users’ personal information.  None of that information was compromised.

How it happened?

If we were hacked, it has nothing to do with PageLines software and probably not WordPress either.

Most likely someone was able to gain some access to the filesystem through a backdoor in a WordPress plugin we were using. They had probably figured this out over the past weeks, and were biding their time until we were close to launch.

Backdoors can  be tough to find and are often well disguised. For this reason we have disabled most plugins, and are checking these systems.

Thanks for your patience and support

We want to say “thanks” to the community including PageLines developers, enthusiasts and especially the members of the ’300′ who have supported us through this.

You guys are the reason we work so hard and your support in social media and blog post comments helped tremendously.

Finally, checkout the #PageLines (or #PLSTRONG community hashtag) to see what people are saying or talk w us. :)

133 comments
bsociallaw
bsociallaw

I am not a developer. I have no time for it anymore and my skills are way dated. I hired someone a two months ago to do the basic work. We are just about to start another round of development. Now you are telling me that I have to pay to have the work re-coded to work with the new product? Wow. Do you think you could alienate your customer base any more?

Please confirm either here or by email that I understand your position correctly:

1. You are discontinuing PL Pro and the older Framework products. 

2. You are introducing a replacement product (DMS).

3. You do not provide an automated path to migrate or upgrade sites developed on previous PL products. 

4. Users that invested hours, days, weeks of development in the Framework and PL Pro sites must now invest their own time and/or money to have their old sites re-engineered to play nice with the new DMS product. 

Assuming you respond in the affirmative... are you kidding me? I trust that you will not be billing any Pro accounts this month? You should be offering a three month refund to existing customers and support for those with migration issues. And that's just for starters. 

I'm not a lawyer but I do manage a law firm. I will be raising this with our team to review legal options. Any other users interested in learning about possible recourse can contact me at the following address: pagelines at bsociallaw dot com. 

Pagelines, if you would like to reach me directly please find our phone number at http://bsociallaw.com or use the email above.

Rory Phimister

Sean Clifford
Sean Clifford

Now that things have settled down a bit and y'all aren't scrambling, what details can you give us about the hack? Specifically, I'd like to know which plugin was exploited and which version. I keep things locked down pretty tightly, use SSL for wp-admin religiously. I'm currently using a multi-site WordPress installation (3.6), a child theme with the PageLines Framework, and will shortly be rolling out BuddyPress.

Knowing the cause of the hack and which plugin was exploited would be lovely. Many thanks.

dwp18
dwp18

I am generally happy with PL.  But I am noticing the following problems:

1.  Page text colors-- when I change color of H1 elements, it is NOT working/ manifesting.

2.  Google API is not working.  I inserted the Google Font I desire, and even included the Google Font style HTML/CSS in the PL Custom Code under header, and it is NOT working.

3.  Post thumbnails are set to appear everywhere but they are NOT being shown.

Invisioning
Invisioning

I'm leaving PL and DMS.

With the level of communication and customer service for those of us who have help them grow, I cannot take the uncertainty of having to re-create all 200+ of the sites we provide service to.

To have us sidelined because of a new product, make it NOT backwards compatible OR provide any migration options other than a complete rebuild, make us flounder with having to FTP the plugins we've purchased and try to guess to the correct locations, barely answering any concerns in the forums and kicking us in the nuts with the "Pay our new price and anything you paid before is gone" is business suicide.   

Best of luck guys.

I'm not putting my business in your hands again.

cookcountyblog
cookcountyblog

I've built and maintained 6 sites for Cook County Government. You might recognize that as being the home of Chicago. Started with Platform Pro. Have had no real complaints until now--2 weeks before I'm supposed to relaunch all 6 sites in Framework.

Now I'm getting problems with style sheets that wont load in Framework. Worked through a problem on one of the new sites yesterday, now it crops up on a different one. will it work properly on launch day? Up until yesterday i was positive. Now Im fearful.

Although I've read on their site that I can continue to use Framework, I'd like to hear something definitive telling me that I'm good to proceed on the platform I've spent 8 months developing sites on.

Someone also talked about a PL pro group that was for hire. If these problems continue, I'd like to hire them. They should quickly be able to fix what it took my vendor and GoDaddy support hours trying to figure out.

artmexx
artmexx

I am very very disappointed about the PL´s Product Policy! Why don't they allow their customers to use the old Framework? They have paid for it. I´ve build many websites with the framework. What can i tell my clients? Never trust in PL again?! So please, responsible people in the PL company: Give us back the choice! If not - i will build my websites with other systems - no matter if the dms might by good ...

Stephen Salstrand
Stephen Salstrand

Where can we contact the Pros?  The page is now gone and I need to hire someone.

cookcountyblog
cookcountyblog

Are any current Framework users having problems loading stylesheets?  All of a sudden one of my stylesheets won't load.  And on another site, somehow I've lost ALL categories from posts.  We've been and still are reviewing with our webhost now; but I understand things are pointing towards Pagelines.

rabergemann
rabergemann

Come on Pagelines - give us a realistic update, for goodness sake!!  I need to access my account from Wordpress but I keep getting the message that I cannot access Pagelines with my username.  Entering Pagelines via the new launchpad link is useless if you are working through Wordpress.  If there is another way to do this - please tell us!   I am really concerned that we are going to be just dumped into DMS and it will not work.  

bierblog
bierblog

hey guys, 

i can't login anymore. saw pagelines changed my status from developer to business and now nothing works anymore...

cookcountyblog
cookcountyblog

At some point of their product development, they need to look at their users, determine who have large networks of sites or extremely large sites and figure out how to either help them migrate.  Or reassure them that the products and businesses that have been built with the most recent product are safe and supported in the launch of the new product.  

I'm not sure if I ever saw an email saying something along the lines of "The Future of Framework after DMS".  I'll admit I may have missed it.  Did anyone else get something along those lines?

Apple and MS support multiple OS's for years after launch.  PL needs to be clear on how closely they will follow that model or not.  

cookcountyblog
cookcountyblog

Still can't login through the site dashboard.  I'm on Framework, and have no thoughts of changing to DMS.  I have found docs where PL will keep Framework up and support it for 18-24 months.  That's fine from where I sit.  I may adopt DMS; but in August, I'd like to launch the 6 sites I've already built in Framework first!

anweb
anweb

Hello, I honestly have to say that I've been a PageLines lover since the moment I discovered it, I think it has been a great tool to me and I am very grateful for what I was able to accomplished with the old framework system, however, this new DMS deal is forcing me to feel differently and a bit unhappy. I have build several websites using the framework and purchased a couple of templates that now are useless cause I can't use them with this new system anymore. I don't know how to tell my clients, "hey the framework I used to build your site, is now gone, so I will have to "recreate" your site design using a new DMS, which btw, was hacked twice". How can I expect my clients to rely on me when I feel I can't rely on PageLines anymore? I'm afraid that if I build new sites using DMS, PL will come up with something else and all my work and clients will be threaten again. I agree with Brethen, I think PL should give us a choice of keeping the old framework intact and migrate to DMS if we so desire, and make the transition smoothly.  I also don't understand why PL destroyed the amazing support it had with the live chat for PageLines Plus ( Dev ), now what it is? A forum? The 300 community? PL please,  If it ain't broke, don't fix it.

MundoCaco
MundoCaco

well,

I reading too long post about patience and service about Pagelines service... people be patience!

For the Pagelines team this is a crucial time about a new direction about the web design, web service and a  big hack in the system.

Everyboy knows that the past PL service  can make a lot of web design by the same price.

Now is time to make a move forward - together.

move on!!

cookcountyblog
cookcountyblog

Cant login my dashboard.  Changed my password 4 times. Would like to launch a new site within the next 7-10 days. Been with Pagelines since 2011.

kwitelle
kwitelle

What a mess, as always with Pagelines. 

Do we now have to pay again to keep using this Framework or should I say the "revolutionary" DMS system??? 

I don't see any revolutionary thing about it. Still the same features, still the same Drag & Drop thing only now it is kinda in your browser window underneath the site and not via the backend of Wordpress. Ok, maybe a bit more handy. But is what is so revolutionary about this?

I also don't understand their pricing system. I has changed again. First you had Pro and Dev, then you had that "Plus" thing and now again something else.

The fucking thing is that when you have about 10 sites running on Pagelines you have to swallow with this crap. 

flidget
flidget

Cor, just spotted the update today in the static text above. It's really easy to miss it if you're active in the comments.

PetraNiedermoser
PetraNiedermoser

Wow. I now have 0 karma points in my account, but no grandfathered subscription. Waited 3 weeks with a Clients Project because of DMS. Habe been patient. Now it's time to say...

Brenden
Brenden

@PageLines  Come  on guys, what are you doing!!!  We are all being very patient.  All we are asking is stop trying to force us into your broken piece of "DMS" and reactivate the old.  Give us a choice!  Let those who want to migrate do so and let the rest of us continue on our merry way.  We understand that this is your world and we are just living in it.  However, don't forget, we purchased land in your little world.  That should entitle us to a little say.

Mictlan
Mictlan

@Invisioning 

Here! here!

I can't put it more clear thank you. We built a business model around framework and since pagelines is not giving us any real reason to stay. I am talking as a pro business client not sure developers think a like. Anyhow cost me the same upgrade now that if I decide to comeback in the future. So its time to rethink why should I put all my eggs in one basket. what happen if the basket decide to change again with any option of choice. But that is just the nature of business. As I mention in the past I am sure this was a calculated decision.

Best of luck.


arpowers
arpowers moderator

@cookcountyblog nothing that way should have broken, everything else should be back to normal by tonight. We've switched the server strategy. 

arpowers
arpowers moderator

@fvncy I appreciate your support, trust me we by no means meant to have any issues w/ v2 this week. We're keeping it around and healthy. 

arpowers
arpowers moderator

@Jack Tummers we're fixing those issues. We'll also work with you to make you happy. Please just communicate and help us create the correct policy

MartinRob
MartinRob

@rabergemann

This is what i have done and it is working for me (well its letting me get on and build sites) go to http://launchpad.pagelines.com then log in you should see PageLines Framework v2.4.4 - Developer.(i am asuming you have a paid PL developer license)

download the zip file. and any other sections you have bought will also be there for you to download and use.

goto wordpress admin

themes

then upload the zip file.


cookcountyblog
cookcountyblog

@rabergemann Amen to that.  I'm working on a launch of 2 big and 4 smaller government sites.  I want to launch the week of 9/12, and I can't migrate everything to DMS.

flidget
flidget

@MundoCaco Actually it's a crucial time for all the people dependent on the products they purchased before DMS. It's not that crucial for Pagelines; they still have all our money from the previous frameworks so they are not going to go out of business for the sake of a month's delay on their launch. Unless they have spent it all. However, some independent contractors may lose income and long term client relationships, which is the really crucial point.

There is a new direction in IT every year, or so the marketers would like you to believe. I wouldn't fall for that hype if I was you. Look back through all these blog posts and try to find the advantages and new features that are not simply an expanded export and a new UI on old ideas. In reality there will be a lot of new code to lock down functionality and restrict usage under the new subscription model.

A real new direction for Pagelines would be to have secure library management, source code control and staging servers for their products. Then there wouldn't even have been such a massive catastrophe.

We all want to move on, we are just waiting for Pagelines so that we can get back to where we were. Hopefully, Pagelines will implement much better (industry standard) source code safeguards and controls to prevent such a massive inconvenience for their existing customers going forward. 

The only people who will benefit from this episode is in fact Pagelines, who should come out of this much more aware of the type of business they are proposing to operate and how much more they need to invest in traditional application development approaches and processes. The system uptime and availability required for a subscription system is very high. Multiple redundancy needs to be in place. Do we really think Pagelines can step up to this higher level of operation? It's obviously not built in from the start, or we don't know the full story behind the issues.

DMS literally isn't going to work on subscription if Pagelines isn't able to keep their service running and get it back at the drop of a hat in case of a disaster. Think hard about this folks, it's painful enough now, but all this code that needs recovering is going to be primarily concerned with making sure you cannot access the new functionality without the subscription option - it's probably not code that gives added functionality and fundamentally it's code with Pagelines' benefit in mind.

This "big hack" was supposed to be on the front end system, not the safe and secure development or test systems? It feels like they don't even have a deployment package. Let's hope going forward they have something simple like a transportable backup and scripted configuration instructions.

Brenden
Brenden

BTW 

By End of Week:

  • Fix on v2 Store API; which will repair v2 store, other issues.
  • v2 Store System,
  • Finishing up grandfathering Plus members into DMS Pro

Brenden
Brenden

@PetraNiedermoser And you were giving me a hard time about my patience the other day lol.  We spent one week in workflow workshops to learn this new system and they dumped it on us 2 weeks later. Grrrrrr  I feel your frustrations... Hang in there... In the mean-while you might want to bust out notepad and start programing the ol' fasioned way ;)

Kyle_n_Irving
Kyle_n_Irving

@Brenden Hi, if you have already purchased the Framework you will be able to find it here http://launchpad.pagelines.com 

If you need support you can go to the forum 
http://forum.pagelines.com

btw DMS is not broken. I currently have 2 sites up using DMS, what i love most about it is the speed i am able to build sites at. So much faster than Framework, as with everything new though it takes a little getting used to. 

cookcountyblog
cookcountyblog

Will I be able to add new sections? Will DMS sections and plugins work with both?

rabergemann
rabergemann

@cookcountyblog @rabergemann I know every company has its problems every now and then, but this is such bad customer service, especially since Pagelines knows we are dependent on their systems for our companies and clients


cookcountyblog
cookcountyblog

@MundoCaco @cookcountyblog   I can sign on from that url.  I can't sign on from the login fields on the "Your Account" tab at the Pagelines section on the Wordpress dashboard.

MundoCaco
MundoCaco

Hello @flidget I very much agree with everything you mention.

In my personal case, this time of 'disaster' or break of Pagelines - and some of my clients - has made me think about how much I depend of Pagelines for my web design services.

In my case, I do not depend 100% and makes me choose other options to continue to give good service to my customers and give a true value of the service provide Pagelines for me.

Behind my comment, what I mean is: how far depend on Pagelines?

I think they know their business, so this made me ​​rethink how is my business and my level of dependency of Pagelines. I need to know that I can choose, always.

This is I think. :)

Thank you!

PetraNiedermoser
PetraNiedermoser

@Brenden It's quite a bit ironic, that we get karma points (from pagelines ;;)) while we wait for using what we've paid for. 

And while we speak of karma... the whole concept of karma does not fit to this attitude. That's not the way I want to make business. If I were a private user, I'd be patient and wait. But I have a dev license and a plus account because I work with clients, and also THEY work with clients. Need a partner who does business like a pro.

Hack or not, the WAY they handle this, has nothing to do with some launch problems. It's about attitude, respect and professionalism.

Sometimes you can't be faster. Because time is time. For all of us. And sometimes unexpected things are going an. Important is the way you handle this things and whether you are able to use the potential or not.

blogmylunch
blogmylunch

@Brenden Do you have an email? I can be contacted at admin@blogmylunch.com or on Twitter or FB.

Brenden
Brenden

@Kyle_n_Irving @Brenden   All new workflows take a little getting used too.  I appreciate the friendly advise on where to find it. I do know where to find it though.

   However, I beg to differ with you and so will over 200 people who have complained about DMS being broken.  I am glad you have 2 sites running on it.  However, my company purchased the "Old Framework" Pro Dev v and we put up over 20 sites in 2 weeks that are now also broken because they have failed to complete a smooth transition.  PL is failing misserably and you will not get any brownie points for sticking up for their failures, you will only get a brown nose.  

  They had a great platform and they dumped it for greed.  I don't blame them for wanting to make more money, we all want that, but too force those who don't want to go to the DMS system is just greed and bad business.

Brenden
Brenden

@PetraNiedermoser @Brenden  Very well put Petra  I could not agree more.  I personally would give you oudles of karma points.  You have been more patient then me, that is for sure.  I had to start taking yoga classes this week just to keep calm lol While we wait and gain "Karma points" for our patience, we look unprofessional to our clients.  I despise liars and lieing and I am not about to use the lame excuse to my clients that PL did with us, "We were hacked...whaa" wait we were hacked again... Come on enough is enough.  Put it up or shut up... I mean how hard is it to just put up the old platform as well.

Jack Tummers
Jack Tummers

@Martin Bay @Kyle_n_Irving @Brenden I'm looking for those free plugins and sections as well. I added a topic about this in the forum yesterday, but no reply yet. I can only see premium plugins, even with my 'Framewok only' account.

BreeReese
BreeReese

@flidget @Kyle_n_Irving @Brenden Wow- they said that? Rob has always been AMAZINGLY helpful to me in my ridiculous bouts of "what the hell am I doing wrong." I'm so very bummed that we have all been essentially forgotten. IT MAKES NO SENSE that the post-snafu fix was to their new system and not all the professionals who depend on them to fulfill their end of the contract we ALL entered into when we gave PL our hard-earned money.

I'm so effing irritated I could spit.

erwanlescop
erwanlescop

@Kyle_n_Irving

Hi Kyle,

I really don't mind using the "new" Framework launchpad vs. dashboard, not a big deal... provided we have all (or at least most of) the products we bought.

Which is unfortunately still not the case atm, or I miss something ;)

As far as I'm aware: 95% of PL v2 - Framework free plugins are not available (except the one and only "
Grid Shortcodes"), many free sections are not available and/or can't be activated, and not a single Plus plugin/section (yes, I'm a Plus susbcriber) is available.

I really hope I'm wrong ;) !

Best regards.

 @flidget

Kyle_n_Irving
Kyle_n_Irving

@flidget @Kyle_n_Irving you are right fidget - we all have our own definitions. 

right now all your products should be in the launchpad.pagelines.com - i know mine are - i know its not as convieniant as using the dashboard but its a solution and it works for now. 

i am not sure what the short term plan is with the dashboard store - i have asked for it to be fixed like everyone is. and i know that they are working on it.  

the store is now in a new infrastructure and they used rss feeds previously  to keep the dashboard up to date. 

this is a small team and there are a lot of things they have to fix. i don't want to come over as an apologist - however - we are builders and we find solutions - and right now - as inconvenient as it is - we have to get our products from the launchpad.

flidget
flidget

@Kyle_n_Irving @Brenden Hi Kyle, when I login to Pagelines I don't see any of my purchases. How do you get to this launchpad you mention. PS bash Lovefilm for me too please.

Kyle_n_Irving
Kyle_n_Irving

@Brenden @Kyle_n_Irving you can get everything you paid for in the launchpad.

yes thats not as convenient as via the dashboard and i am frustrated about that as well, however,  posturing on here is not going to help. better channels are email or forum. and if you really want a refund - email is the only route.

and if you want to hear me complain just tune into the call lovefilm is about to get... 

flidget
flidget

@Kyle_n_Irving @Brenden Hi Kyle, hope you nose is quickly returning to normal colour :-) One man's definition of broken is allowed to contain the fact you cannot currently make any enhancements to the exiting sites in terms of new or improved sections etc. I fully support this view which is a simple premise, you cannot do now what you could do when it was working - so it's broken. Also, there is no obvious action underway to get it working again or any explanation of how deleting static files as mentioned in the posts about the hack caused this problem in the first place. On the up side, as we all sit around waiting on Pagelines to fix their product we have plenty of time to bash away on this blog.

---snip from the support forum---

Rob, on 28 Jul 2013 - 4:11 PM, said:

Willem, the only thing I can say is it will be fixed when it is fixed.  We have no timeline or estimated time of arrival.  I apologize for the inconvenience to you and your clients.  We all just have to wait and watch as our development team fixes one bit at a time.

Brenden
Brenden

@Kyle_n_Irving@BrendenNo my intention is not in insult.  It is to point out that sticking up for PL is not going to get things fixed.  It simply sends a message of tollerance.  A message that says it is ok to screw those who have already paid good money and just want to go back to doing what we do best and that is not complaining about there lack on forums.  We just want to get back to programming.  

We paid for convenience and your statement,  "just can't access the pagelines dashboard at the moment" is counterdictive.  The fact that none of us "just can't access the pagelines dashboard at the moment" denotes that it is broken.  This is a big inconvenience when we are trying to access our store and prior purchases, sections, plugins, etc.

Your happy, that is great, I am happy that you are happy.  However, if I were like you I would still be stuck with a pair of shoes I bought online that fell apart.  No, I want and get what I paid for or I get refunded, as we all should.

Kyle_n_Irving
Kyle_n_Irving

@Brenden  there is no reason why your Framework sites would be broken - i have 20+ they are all running fine - just can't access the pagelines dashboard at the moment but i can get everything i need from launchpad.

i even have 3 pagelines pro sites and they are still running fine. 

as to your comment regarding the colour of my nose - that is very personal, however,  i am sure your intention was not to insult. Thanks